Skip to content

NPE in JDBC Source using query #560

Open
@rmoff

Description

@rmoff
Member

Source object

mysql> describe transactions;
+---------------+--------------+------+-----+---------+-------+
| Field         | Type         | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| txn_id        | int(11)      | YES  |     | NULL    |       |
| customer_id   | int(11)      | YES  |     | NULL    |       |
| amount        | decimal(5,2) | YES  |     | NULL    |       |
| currency      | varchar(50)  | YES  |     | NULL    |       |
| txn_timestamp | varchar(50)  | YES  |     | NULL    |       |
+---------------+--------------+------+-----+---------+-------+
5 rows in set (0.01 sec)

Config:

[2019-01-04 17:09:40,225] INFO JdbcSourceTaskConfig values:
 batch.max.rows = 100
 catalog.pattern = null
 connection.attempts = 3
 connection.backoff.ms = 10000
 connection.password = [hidden]
 connection.url = jdbc:mysql://mysql:3306/demo
 connection.user = connect_user
 dialect.name =
 incrementing.column.name = txn_id
 mode = incrementing
 numeric.mapping = null
 numeric.precision.mapping = false
 poll.interval.ms = 5000
 query = SELECT customer_id, amount, currency, txn_timestamp FROM demo.transactions
 schema.pattern = null
 table.blacklist = []
 table.poll.interval.ms = 60000
 table.types = [TABLE]
 table.whitelist = []
 tables = []
 timestamp.column.name = []
 timestamp.delay.interval.ms = 0
 topic.prefix = mysql-xx
 validate.non.null = false
 (io.confluent.connect.jdbc.source.JdbcSourceTaskConfig)

Error:

[2019-01-04 17:09:40,625] ERROR WorkerSourceTask{id=jdbc_source_mysql_xx-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
 at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractOffsetIncrementedId(TimestampIncrementingCriteria.java:231)
 at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractValues(TimestampIncrementingCriteria.java:189)
 at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:185)
 at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:309)
 at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:244)
 at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:220)
 at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
 at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Activity

rmoff

rmoff commented on Jan 4, 2019

@rmoff
MemberAuthor

This was caused by omitting the incrementing.column.name (txn_id) from the SELECT list. With this added back in, the connector worked:

[2019-01-04 17:12:26,496] INFO JdbcSourceTaskConfig values:
 batch.max.rows = 100
 catalog.pattern = null
 connection.attempts = 3
 connection.backoff.ms = 10000
 connection.password = [hidden]
 connection.url = jdbc:mysql://mysql:3306/demo
 connection.user = connect_user
 dialect.name =
 incrementing.column.name = txn_id
 mode = incrementing
 numeric.mapping = null
 numeric.precision.mapping = false
 poll.interval.ms = 5000
 query = SELECT txn_id, customer_id, amount, currency, txn_timestamp FROM demo.transactions
 schema.pattern = null
 table.blacklist = []
 table.poll.interval.ms = 60000
 table.types = [TABLE]
 table.whitelist = []
 tables = []
 timestamp.column.name = []
 timestamp.delay.interval.ms = 0
 topic.prefix = mysql-xx
 validate.non.null = false
 (io.confluent.connect.jdbc.source.JdbcSourceTaskConfig)
rmoff

rmoff commented on Jan 4, 2019

@rmoff
MemberAuthor

Related: #561

vsood-p8p

vsood-p8p commented on Jan 28, 2020

@vsood-p8p

I am getting a similar NPE, but my query has an inner query (due to the where clause in my query).

Here is the config where i try to select the incremental column id and the other config values.

query= select events.evtId, events.* from (select ES.id as evtId, ES.action, U.name FROM eventsource as ES INNER JOIN user U ON ES.user_id = U.id where U.email='xxx') events
mode=incrementing
incrementing.column.name=events.eventId
quote.sql.identifiers=never

Can you please the appriate syntax to overcome this error.

Following is the stacktrace:

java.lang.NullPointerException
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractOffsetIncrementedId(TimestampIncrementingCriteria.java:237)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractValues(TimestampIncrementingCriteria.java:195)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:192)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:316)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:204)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-01-28 14:14:37,251] ERROR WorkerSourceTask{id=connect-event-source1-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:173)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @rmoff@vsood-p8p

        Issue actions

          NPE in JDBC Source using query · Issue #560 · confluentinc/kafka-connect-jdbc