Open
Description
Source object
mysql> describe transactions;
+---------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+-------+
| txn_id | int(11) | YES | | NULL | |
| customer_id | int(11) | YES | | NULL | |
| amount | decimal(5,2) | YES | | NULL | |
| currency | varchar(50) | YES | | NULL | |
| txn_timestamp | varchar(50) | YES | | NULL | |
+---------------+--------------+------+-----+---------+-------+
5 rows in set (0.01 sec)
Config:
[2019-01-04 17:09:40,225] INFO JdbcSourceTaskConfig values:
batch.max.rows = 100
catalog.pattern = null
connection.attempts = 3
connection.backoff.ms = 10000
connection.password = [hidden]
connection.url = jdbc:mysql://mysql:3306/demo
connection.user = connect_user
dialect.name =
incrementing.column.name = txn_id
mode = incrementing
numeric.mapping = null
numeric.precision.mapping = false
poll.interval.ms = 5000
query = SELECT customer_id, amount, currency, txn_timestamp FROM demo.transactions
schema.pattern = null
table.blacklist = []
table.poll.interval.ms = 60000
table.types = [TABLE]
table.whitelist = []
tables = []
timestamp.column.name = []
timestamp.delay.interval.ms = 0
topic.prefix = mysql-xx
validate.non.null = false
(io.confluent.connect.jdbc.source.JdbcSourceTaskConfig)
Error:
[2019-01-04 17:09:40,625] ERROR WorkerSourceTask{id=jdbc_source_mysql_xx-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NullPointerException
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractOffsetIncrementedId(TimestampIncrementingCriteria.java:231)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractValues(TimestampIncrementingCriteria.java:189)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:185)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:309)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:244)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:220)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Activity
rmoff commentedon Jan 4, 2019
This was caused by omitting the
incrementing.column.name
(txn_id
) from theSELECT
list. With this added back in, the connector worked:rmoff commentedon Jan 4, 2019
Related: #561
vsood-p8p commentedon Jan 28, 2020
I am getting a similar NPE, but my query has an inner query (due to the where clause in my query).
Here is the config where i try to select the incremental column id and the other config values.
query= select events.evtId, events.* from (select ES.id as evtId, ES.action, U.name FROM eventsource as ES INNER JOIN user U ON ES.user_id = U.id where U.email='xxx') events
mode=incrementing
incrementing.column.name=events.eventId
quote.sql.identifiers=never
Can you please the appriate syntax to overcome this error.
Following is the stacktrace:
java.lang.NullPointerException
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractOffsetIncrementedId(TimestampIncrementingCriteria.java:237)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.extractValues(TimestampIncrementingCriteria.java:195)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:192)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:316)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:204)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-01-28 14:14:37,251] ERROR WorkerSourceTask{id=connect-event-source1-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:173)