weixin_39927683
weixin_39927683
2021-01-02 10:00

Logstash 6.2.1 and jdbc 4.3.3 Java::JavaSql::SQLException: No suitable driver found for jdbc:sybase:Tds:server:port

Hello, I'm experiencing issue with input jdbc plugin after upgrade to logstash 6.2.1 or 6.2.0 from 6.1.3. Same config, same setting, only different logstash package. If I downgrade the logstash everything works fine again.

Thank you for your help

  • Version: Logstash 6.2.1 and jdbc 4.3.3
  • Operating System: CentOS 7
  • Config File:

input {

    jdbc {
    id => "xxx"

    jdbc_driver_library => "/etc/logstash/syb-jconn.jar"
    jdbc_driver_class => "com.sybase.jdbc4.jdbc.SybDriver"
    jdbc_default_timezone => "Europe/Prague"

    jdbc_connection_string => "jdbc:sybase:Tds:server:port"
    jdbc_user => "xxx"
    jdbc_password => "xxx"

    add_field => { "server" => "xxx" }
    last_run_metadata_path => "/var/lib/logstash/metadata/xxx"
    schedule => "3-59/5 * * * *"
    tracking_column => "eventtime"
    use_column_value => "true"
    tracking_column_type => "timestamp"
    statement => "xxx"
  }
  • Sample Data: /var/log/logstash/: [ERROR][logstash.inputs.jdbc ] Unable to connect to database. Tried 1 times {:error_message=>"Java::JavaSql::SQLException: No suitable driver found for jdbc:sybase:Tds:server:port"}

journalctl:


rufus-scheduler intercepted an error:
job:
  Rufus::Scheduler::CronJob "0-59/3 * * * *" {}
error:
  2030
  Sequel::DatabaseConnectionError
  Java::JavaSql::SQLException: No suitable driver found for jdbc:sybase:Tds:server:port
 java.sql.DriverManager.getConnection(java/sql/DriverManager.java:689)
 java.sql.DriverManager.getConnection(java/sql/DriverManager.java:247)
 java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
 org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
 org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:355)
 RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/adapters/jdbc.rb:191)
 RUBY.make_new(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/connection_pool.rb:126)
 RUBY.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/connection_pool/threaded.rb:192)
 RUBY.acquire(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/connection_pool/threaded.rb:133)
 RUBY.hold(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/connection_pool/threaded.rb:90)
 RUBY.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/database/connecting.rb:269)
 RUBY.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/database/connecting.rb:279)
 RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/database/connecting.rb:58)
 RUBY.connect(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/sequel-5.4.0/lib/sequel/core.rb:116)
 RUBY.block in jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.3/lib/logstash/plugin_mixins/jdbc.rb:112)
 org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1292)
 RUBY.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.3/lib/logstash/plugin_mixins/jdbc.rb:161)
 RUBY.execute_statement(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.3/lib/logstash/plugin_mixins/jdbc.rb:227)
 RUBY.execute_query(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.3/lib/logstash/inputs/jdbc.rb:271)
 RUBY.block in run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.3/lib/logstash/inputs/jdbc.rb:250)
 org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
 org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)
 org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)
 RUBY.do_call(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234)
 RUBY.do_trigger(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258)
 RUBY.block in start_work_thread(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300)
 org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
 org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)
 org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)
 RUBY.block in start_work_thread(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299)
 org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1292)
 org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)
 RUBY.block in start_work_thread(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289)
 org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
 org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
 java.lang.Thread.run(java/lang/Thread.java:748)
tz:
  ENV['TZ']:
  Time.now: 2018-02-08 17:20:00 +0100
scheduler:
  object_id: 2014
  opts:
 {:max_work_threads=>1}
 frequency: 0.3
 scheduler_lock: #<:scheduler::nulllock:0x2ed725aa>
 trigger_lock: #<:scheduler::nulllock:0xae4f313>
  uptime: 308.93567199999995 (5m8s935)
  down?: false
  threads: 2
 thread: #<0x25b84145>
 thread_key: rufus_scheduler_2014
 work_threads: 1
active: 1
vacant: 0
max_work_threads: 1
 mutexes: {}
  jobs: 1
 at_jobs: 0
 in_jobs: 0
 every_jobs: 0
 interval_jobs: 0
 cron_jobs: 1
  running_jobs: 1
  work_queue: 0
0x25b84145></:scheduler::nulllock:0xae4f313></:scheduler::nulllock:0x2ed725aa>
  • Steps to Reproduce: Nothing special, just start logstash.

该提问来源于开源项目:logstash-plugins/logstash-input-jdbc

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

23条回答

  • weixin_39773817 weixin_39773817 4月前

    -brownbear - i think it is an issue with Java entry point, but not for the reasons originally thought. MySql and Postgresql work without issue, but Redshift fails when launched with the Java entry point, but works if re-introducing the Ruby entry point.

    I _think the difference is the JRuby TLS artifacts in use by the process when run via the Java entry entry point vs. the Ruby entry point. The diffs of the jars in use by the process i believe is the most telling. However, not sure how to proceed here.

    Could you take a look and the testing below to see what conclusions you draw ?

    Testing environment Logstash 6.2.3 / Fedora 24

    In both cases I don't actually have the DB running, and am watching the error logged. It should load the driver, then complain about the connection, so the error should come from the driver. If the error comes from the driver, I consider it a success...if the error comes from unable to load type error, i consider it a failure (and inline with the OP's issue)

    I added this script to get the ruby entry point back for testing:

    
    #!/usr/bin/env bash
    
    unset CDPATH
    
    . "$(cd `dirname $0`/..; pwd)/bin/logstash.lib.sh"
    setup
    
    ruby_exec "${LOGSTASH_HOME}/lib/bootstrap/environment.rb" "logstash/runner.rb" "$@"
    

    Redshift:

    Works with Ruby entry point, Fails with Java endpoint.

    Config:

    
    input {
      jdbc {
    
        #redshift
        jdbc_driver_library => "/home/jake/RedshiftJDBC42-1.2.12.1017.jar"
        jdbc_driver_class => "com.amazon.redshift.jdbc42.Driver"
        jdbc_connection_string => "jdbc:redshift://examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com:5439/dev"
    
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        schedule => "* * * * * *"
        statement => "SELECT * FROM public.test_table"
        }
    }
    
    output {    
        stdout { codec => rubydebug}
    }
    

    Ruby entry point (works)

    
    Sequel::DatabaseConnectionError
    Java::JavaSql::SQLException: [Amazon](500150) Error setting/closing connection: UnknownHostException.
      com.amazon.redshift.client.PGClient.connect(com/amazon/redshift/client/PGClient)
      com.amazon.redshift.client.PGClient.<init>(com/amazon/redshift/client/PGClient)
      com.amazon.redshift.core.PGJDBCConnection.connect(com/amazon/redshift/core/PGJDBCConnection)
      com.amazon.jdbc.common.BaseConnectionFactory.doConnect(com/amazon/jdbc/common/BaseConnectionFactory)
      com.amazon.jdbc.common.AbstractDriver.connect(com/amazon/jdbc/common/AbstractDriver)
    </init>

    Find the actual Jars for the process:

    
    ps -ef | grep logstash | grep -v grep |  gawk '{ print $2 }' | xargs -n 2 sh -c 'cat /proc/$0/maps' | grep jar
    
    
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/janino-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jsr305-1.3.9.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/logstash-core.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/concurrent-ruby-1.0.5-java/lib/concurrent_ruby_ext.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/jrjackson-0.4.5-java/lib/jrjackson/jars/jrjackson-1.2.23.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-annotations/2.9.4/jackson-annotations-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.9.4/jackson-core-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.4/jackson-databind-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.4/jackson-module-afterburner-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/manticore/manticore-ext.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/puma_http11.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/thread_safe-0.3.6-java/lib/thread_safe/jruby_cache_backend.jar
    /home/jake/logstash-6.2.3/vendor/jruby/lib/jruby.jar
    /home/jake/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/jopenssl.jar
    /home/jake/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/org/bouncycastle/bcpkix-jdk15on/1.56/bcpkix-jdk15on-1.56.jar
    /home/jake/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/org/bouncycastle/bcprov-jdk15on/1.56/bcprov-jdk15on-1.56.jar
    /home/jake/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/org/yaml/snakeyaml/1.18/snakeyaml-1.18.jar
    /home/jake/logstash-6.2.3/vendor/jruby/lib/ruby/stdlib/psych.jar
    /home/jake/RedshiftJDBC42-1.2.12.1017.jar
    /tmp/jar_cache2564280523715558384.tmp (deleted)
    /tmp/jar_cache2690174455721520573.tmp (deleted)
    /tmp/jar_cache3116821306932265205.tmp (deleted)
    /tmp/jar_cache3517657113631923153.tmp (deleted)
    /tmp/jar_cache4431046173424909287.tmp (deleted)
    /tmp/jar_cache4764733334724072251.tmp (deleted)
    /tmp/jar_cache5250361652349903858.tmp (deleted)
    /tmp/jar_cache5644978546018668552.tmp (deleted)
    /tmp/jar_cache7426803057865609509.tmp (deleted)
    /tmp/jar_cache7713999098356712899.tmp (deleted)
    /tmp/jar_cache8702422410265496312.tmp (deleted)
    /tmp/jar_cache8830396966131713336.tmp (deleted)
    /tmp/jar_cache908373400556636540.tmp (deleted)
    /usr/java/jdk1.8.0_144/jre/lib/ext/nashorn.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/sunec.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/sunjce_provider.jar
    /usr/java/jdk1.8.0_144/jre/lib/jce.jar
    /usr/java/jdk1.8.0_144/jre/lib/jsse.jar
    /usr/java/jdk1.8.0_144/jre/lib/resources.jar
    /usr/java/jdk1.8.0_144/jre/lib/rt.jar
    

    Java entry point (doesn't work)

    Note - the error is not coming from the driver JAR, and assuming that the driver JAR can not load.

    
    Sequel::DatabaseConnectionError
    Java::JavaSql::SQLException: No suitable driver found for jdbc:redshift://examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com:5439/dev
      java.sql.DriverManager.getConnection(java/sql/DriverManager.java:689)
      java.sql.DriverManager.getConnection(java/sql/DriverManager.java:247)
      java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
      org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:423)
      org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:355)
      RUBY.connect(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/adapters/jdbc.rb:191)
      RUBY.make_new(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/connection_pool.rb:126)
      RUBY.assign_connection(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/connection_pool/threaded.rb:192)
      RUBY.acquire(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/connection_pool/threaded.rb:133)
      RUBY.hold(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/connection_pool/threaded.rb:90)
      RUBY.synchronize(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/database/connecting.rb:269)
      RUBY.test_connection(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/database/connecting.rb:279)
      RUBY.connect(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/database/connecting.rb:58)
      RUBY.connect(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/sequel-5.5.0/lib/sequel/core.rb:116)
      RUBY.block in jdbc_connect(/home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-jdbc-4.3.5/lib/logstash/plugin_mixins/jdbc.rb:114)
    

    Find the actual Jars for the process:

    
    ps -ef | grep logstash | grep -v grep |  gawk '{ print $2 }' | xargs -n 2 sh -c 'cat /proc/$0/maps' | grep jar
    
    
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/janino-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jsr305-1.3.9.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/logstash-core.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/concurrent-ruby-1.0.5-java/lib/concurrent_ruby_ext.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/jrjackson-0.4.5-java/lib/jrjackson/jars/jrjackson-1.2.23.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-annotations/2.9.4/jackson-annotations-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.9.4/jackson-core-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.4/jackson-databind-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.4/jackson-module-afterburner-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/manticore/manticore-ext.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/puma_http11.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/thread_safe-0.3.6-java/lib/thread_safe/jruby_cache_backend.jar
    /home/jake/RedshiftJDBC42-1.2.12.1017.jar
    /tmp/jar_cache1318987372763823602.tmp (deleted)
    /tmp/jar_cache1716504379030863785.tmp (deleted)
    /tmp/jar_cache233530037634925310.tmp (deleted)
    /tmp/jar_cache33640091220453604.tmp (deleted)
    /tmp/jar_cache3631679155127560560.tmp (deleted)
    /tmp/jar_cache3740556643083083797.tmp (deleted)
    /tmp/jar_cache397148585142821475.tmp (deleted)
    /tmp/jar_cache5050997979033218883.tmp (deleted)
    /tmp/jar_cache6217668939172258996.tmp (deleted)
    /tmp/jar_cache6508072306675631703.tmp (deleted)
    /tmp/jar_cache7557800460402656052.tmp (deleted)
    /tmp/jar_cache898932492834641515.tmp (deleted)
    /tmp/jar_cache9223308874031939730.tmp (deleted)
    /tmp/jruby-19472/jruby1302748357755995070snakeyaml-1.18.jar
    /tmp/jruby-19472/jruby1867302391824992424bcpkix-jdk15on-1.56.jar
    /tmp/jruby-19472/jruby3775418141503729272bcprov-jdk15on-1.56.jar
    /tmp/jruby-19472/jruby832430896524098067psych.jar
    /tmp/jruby-19472/jruby8674492003959704456jopenssl.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/nashorn.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/sunec.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/sunjce_provider.jar
    /usr/java/jdk1.8.0_144/jre/lib/jce.jar
    /usr/java/jdk1.8.0_144/jre/lib/jsse.jar
    /usr/java/jdk1.8.0_144/jre/lib/resources.jar
    /usr/java/jdk1.8.0_144/jre/lib/rt.jar
    

    Diff of Jars for running process:

    image

    Postgresql:

    Works with Ruby entry point and Java endpoint.

    
    input {
      jdbc {
    
        jdbc_driver_library => "/home/jake/postgresql-42.2.2.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/test1"
    
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        schedule => "* * * * * *"
        statement => "SELECT * FROM public.test_table"
        }
    }
    
    output {    
        stdout { codec => rubydebug}
    }
    

    Ruby entry point (works)

    
    Mar 23, 2018 12:27:13 PM org.postgresql.Driver connect
    SEVERE: Connection error: 
    org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:245)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:423)
    </init>

    Java entry point (works)

    Note - same error as Ruby entry point

    
    Mar 23, 2018 12:25:58 PM org.postgresql.Driver connect
    SEVERE: Connection error: 
    org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:245)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
        at org.postgresql.Driver.makeConnection(Driver.java:452)
        at org.postgresql.Driver.connect(Driver.java:254)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:468)
    </init>

    Find the actual Jars for the process:

    
    ps -ef | grep logstash | grep -v grep |  gawk '{ print $2 }' | xargs -n 2 sh -c 'cat /proc/$0/maps' | grep jar
    
    
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/janino-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jsr305-1.3.9.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/logstash-core.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/concurrent-ruby-1.0.5-java/lib/concurrent_ruby_ext.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/jrjackson-0.4.5-java/lib/jrjackson/jars/jrjackson-1.2.23.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-annotations/2.9.4/jackson-annotations-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.9.4/jackson-core-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.4/jackson-databind-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/vendor/jar-dependencies/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.4/jackson-module-afterburner-2.9.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/manticore/manticore-ext.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/puma_http11.jar
    /home/jake/logstash-6.2.3/vendor/bundle/jruby/2.3.0/gems/thread_safe-0.3.6-java/lib/thread_safe/jruby_cache_backend.jar
    /home/jake/postgresql-42.2.2.jar
    /tmp/jruby-18284/jruby3088585496297461962bcpkix-jdk15on-1.56.jar
    /tmp/jruby-18284/jruby6903380553027994176snakeyaml-1.18.jar
    /tmp/jruby-18284/jruby7800530674962736051bcprov-jdk15on-1.56.jar
    /tmp/jruby-18284/jruby7864208556168283001psych.jar
    /tmp/jruby-18284/jruby8157230096742516637jopenssl.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/cldrdata.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/localedata.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/nashorn.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/sunec.jar
    /usr/java/jdk1.8.0_144/jre/lib/ext/sunjce_provider.jar
    /usr/java/jdk1.8.0_144/jre/lib/jce.jar
    /usr/java/jdk1.8.0_144/jre/lib/jsse.jar
    /usr/java/jdk1.8.0_144/jre/lib/resources.jar
    /usr/java/jdk1.8.0_144/jre/lib/rt.jar
    

    Difference : image

    Looking at the class path from inside the Ruby pipeline

    Add the following to ./logstash-core/lib/logstash/pipeline.rb

    
    start_flusher # Launches a non-blocking thread for flush events
    
    puts("****** System Class Loader ********")
    cl = java.lang.ClassLoader.getSystemClassLoader().to_java(java.net.URLClassLoader)
    urls = cl.getURLs()
    urls.each do |url|
      puts(url.getFile())
    end
    puts("****** END System Class Loader ********")
    
    wait_inputs
    

    Ruby entry point

    postgresql config

    
    ****** System Class Loader ********
    /home/jake/logstash-6.2.3/
    /home/jake/logstash-6.2.3/
    ****** END System Class Loader ********
    

    Redshift config

    
    ****** System Class Loader ********
    /home/jake/logstash-6.2.3/
    /home/jake/logstash-6.2.3/
    ****** END System Class Loader ********
    

    Java entry point

    postgresql config

    
    ****** System Class Loader ********
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/janino-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jsr305-1.3.9.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/logstash-core.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    ****** END System Class Loader ********
    

    Redshift config

    
    ****** System Class Loader ********
    [2018-03-23T13:24:45,449][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<0x61e3175e run>"}
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/janino-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jsr305-1.3.9.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/logstash-core.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    ****** END System Class Loader ********
    </0x61e3175e>

    Moving the driver to the class path

    cp ../RedshiftJDBC42-1.2.12.1017.jar logstash-core/lib/jars/

    Java entry point for Redshift driver on the classpath (works)

    This does indeed fix it

    
    Sequel::DatabaseConnectionError
    Java::JavaSql::SQLException: [Amazon](500150) Error setting/closing connection: UnknownHostException.
      com.amazon.redshift.client.PGClient.connect(com/amazon/redshift/client/PGClient)
      com.amazon.redshift.client.PGClient.<init>(com/amazon/redshift/client/PGClient)
      com.amazon.redshift.core.PGJDBCConnection.connect(com/amazon/redshift/core/PGJDBCConnection)
      com.amazon.jdbc.common.BaseConnectionFactory.doConnect(com/amazon/jdbc/common/BaseConnectionFactory)
      com.amazon.jdbc.common.AbstractDriver.connect(com/amazon/jdbc/common/AbstractDriver)
      com.amazon.redshift.jdbc.Driver.connect(com/amazon/redshift/jdbc/Driver)
      java.sql.DriverManager.getConnection(java/sql/DriverManager.java:664)
      java.sql.DriverManager.getConnection(java/sql/DriverManager.java:247)
    </init>

    ..and the class path from inside

    
    ****** System Class Loader ********
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/google-java-format-1.5.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/guava-22.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/janino-3.0.8.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/jsr305-1.3.9.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/logstash-core.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/RedshiftJDBC42-1.2.12.1017.jar
    /home/jake/logstash-6.2.3/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    ****** END System Class Loader ********
    
    

    ... and the diff to the java entry point when it is not on the classpath: image

    Huh ?

    Still not sure what is actually happening... An educated guess is that the lack of the some the JRuby TLS artifacts (see diff screen shots) is the root cause. I presume that Redshift (and the others mentioned in this thread) rely on some of the TLS artifacts possibly in static scope and is failing to load. I also assume that when the driver is on the proper Java class path it can use the Java's runtime TLS stuff.

    I did try to add compile "org.jruby:jruby-stdlib:${jrubyVersion}" to logstash core ... but it didn't seem to make a difference (alot of the missing jars are provided scoped transitive dependencies)....and pretty sure that is bad idea. Perhaps we need to explicitly add those missing jars to logstash core's class path ?

    image

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    I wasn't yet able to reproduce this, but I'm pretty convinced you found the issue here.

    I think you could simply revert ls core to the old behavior by loading jar dependencies and ruby maven in LS core, before plugins are loaded.

    You can do this by reverting this block (and only this block, not the eval below) https://github.com/elastic/logstash/pull/8161/files#diff-a70557553ce36f52ecddde61c83cb0f4L73. If you should randomly find some time to try out if that fixes things and still have everything set up to do so -> would be great if you could give it a go. Otherwise, I can try to set this reproducer up as well :)

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    I tried doing what I suggested above. It didn't work, unfortunately.

    Next thing I'd try would be to manually add jruby-openssl to the gemspec of this plugin. That should restore the old load order I guess?

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    -brownbear - tried adding jruby-openssl to the gemspec, but no luck

    lockfile for Logstash

    
    PATH
      remote: /Users/jake/workspace/plugins/logstash-input-jdbc
      specs:
        logstash-input-jdbc (4.3.6)
          jruby-openssl
    ...
    

    Same "No suitable driver found for ..." problem.

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    It seems that we can fix this by explicitly adding the driver to the system class loader, instead of the JRuby runtime class loader.

    We currently load the jar with this directive:

    
    require driver
    

    Which I think just does this :

    
      class_loader = JRuby.runtime.jruby_class_loader
      class_loader.add_url(java.io.File.new(driver).toURI().toURL())
    

    at least the behavior, and contents of that class loader identical. ...but if i change this to

    
       class_loader = java.lang.ClassLoader.getSystemClassLoader().to_java(java.net.URLClassLoader)
       class_loader.add_url(java.io.File.new(driver).toURI().toURL())
    

    it seems to work.

    -brownbear - thoughts on this simple change ?

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    first of all, 1000x thanks for figuring that out! :)

    I'm not so sure about the fix though. This wouldn't necessarily be BwC would it? It seems to me like it would be preferable if we could change the classloader behavior for the new entry point to more closely mimic what the JRuby provided Main entry point does for the classloading? Otherwise, we basically broke the require JRuby call for all plugins (in this specific usage only obv, but still?) and don't fix it?

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    -brownbear - I would have to test for BwC.

    I am not sure how we can keep the java entry point and make it behave the same as prior. A quick examination of the two class loaders against released versions:

    In runner.rb

    
          puts("****** System Class Loader ********")
            class_loader = java.lang.ClassLoader.getSystemClassLoader().to_java(java.net.URLClassLoader)
            urls = class_loader.getURLs()
            urls.each do |url|
              puts(url.getFile())
            end
          puts("****** END System Class Loader ********")
    
    
          puts("****** JRuby Class Loader ********")
            class_loader = JRuby.runtime.jruby_class_loader
            urls = class_loader.getURLs()
            urls.each do |url|
              puts(url.getFile())
            end
          puts("****** END JRuby Class Loader ********")
    
    

    6.2.0

    
    ****** System Class Loader ********
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/commons-compiler-3.0.8.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/google-java-format-1.5.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/guava-22.0.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/jackson-annotations-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/jackson-core-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/jackson-databind-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/janino-3.0.8.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/javac-shaded-9-dev-r4023-3.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/jsr305-1.3.9.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/log4j-api-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/log4j-core-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/logstash-core.jar
    /Users/jake/workspace/releases/logstash-6.2.0/logstash-core/lib/jars/slf4j-api-1.7.25.jar
    ****** END System Class Loader ********
    ****** JRuby Class Loader ********
    /var/folders/x4/76h2pqn96jl492qycz3fz7q00000gn/T/jruby-86831/jruby4618503420881478509psych.jar
    /Users/jake/.m2/repository/org/yaml/snakeyaml/1.18/snakeyaml-1.18.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/concurrent-ruby-1.0.5-java/lib/concurrent_ruby_ext.jar
    /var/folders/x4/76h2pqn96jl492qycz3fz7q00000gn/T/jruby-86831/jruby8391606692301548450bcpkix-jdk15on-1.56.jar
    /var/folders/x4/76h2pqn96jl492qycz3fz7q00000gn/T/jruby-86831/jruby8715141613692041416bcprov-jdk15on-1.56.jar
    /var/folders/x4/76h2pqn96jl492qycz3fz7q00000gn/T/jruby-86831/jruby9173989045272265862jopenssl.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.9.1/jackson-databind-2.9.1.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.9.1/jackson-annotations-2.9.1.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.1/jackson-module-afterburner-2.9.1.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.1/jackson-core-2.9.1.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/jrjackson-0.4.4-java/lib/jrjackson/jars/jrjackson-1.2.22.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/puma_http11.jar
    /Users/jake/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.jar
    /Users/jake/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar
    /Users/jake/workspace/releases/logstash-6.2.0/vendor/bundle/jruby/2.3.0/gems/manticore-0.6.1-java/lib/org/manticore/manticore-ext.jar
    ****** END JRuby Class Loader ********
    

    5.6.7

    
    ****** System Class Loader ********
    /Users/jake/workspace/releases/logstash-5.6.7/
    /Users/jake/workspace/releases/logstash-5.6.7/
    ****** END System Class Loader ********
    ****** JRuby Class Loader ********
    /Users/jake/.m2/repository/org/apache/logging/log4j/log4j-core/2.6.2/log4j-core-2.6.2.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.9.1/jackson-databind-2.9.1.jar
    /Users/jake/.m2/repository/org/apache/logging/log4j/log4j-api/2.6.2/log4j-api-2.6.2.jar
    /Users/jake/.m2/repository/org/slf4j/slf4j-api/1.7.21/slf4j-api-1.7.21.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.9.1/jackson-annotations-2.9.1.jar
    /Users/jake/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.1/jackson-module-afterburner-2.9.1.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.9.1/jackson-dataformat-cbor-2.9.1.jar
    /Users/jake/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.1/jackson-core-2.9.1.jar
    /Users/jake/workspace/releases/logstash-5.6.7/logstash-core/lib/logstash-core/logstash-core.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.5-java/lib/concurrent_ruby_ext.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/jruby/lib/ruby/shared/org/bouncycastle/bcpkix-jdk15on/1.55/bcpkix-jdk15on-1.55.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/jruby/lib/ruby/shared/org/bouncycastle/bcprov-jdk15on/1.55/bcprov-jdk15on-1.55.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/jruby/lib/ruby/shared/jopenssl.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/jrjackson-0.4.4-java/lib/jrjackson/jars/jrjackson-1.2.22.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/puma-2.16.0-java/lib/puma/puma_http11.jar
    /Users/jake/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.jar
    /Users/jake/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpclient/4.5.2/httpclient-4.5.2.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/org/apache/httpcomponents/httpcore/4.4.4/httpcore-4.4.4.jar
    /Users/jake/workspace/releases/logstash-5.6.7/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/org/manticore/manticore-ext.jar
    ****** END JRuby Class Loader ********
    

    I should also note that it is surprising that we we reference the .m2 jars ...nothing to do with this issue, but it seems wrong since we ship the jars in the gems.

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    I think the .m2 is a result of us loading ruby-maven.

    That said, I remember there was an option in JRuby to set the classloader. I guess it's worth looking into that? Will see what I can find out there.

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    A quick looks like the change [1] it is backward compatible, which makes sense if the JRuby class loader is the child of the system loader.

    I agree that we should find a more common way via a core change if one exists, but i think this is viable alternative specific to the JDBC input plugin.

    [1]

    
     #require driver
     class_loader = java.lang.ClassLoader.getSystemClassLoader().to_java(java.net.URLClassLoader)
     class_loader.add_url(java.io.File.new(driver).toURI().toURL())
    
    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    yea that makes sense with the child/parent relation => I'm +1 on fixing it this way for now :)

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    4.3.7 has been released which should resolve this issue.

    
    bin/logstash-plugin update logstash-input-jdbc
    

    Would love to get confirmation of the fix.

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    that said, we should still investigate this ... maybe the jar gem is loaded/configured in some different way now ... looking at https://github.com/logstash-plugins/logstash-input-jdbc/issues/263#issuecomment-368394516 this is pretty likely. Missed that comment before posting above

    点赞 评论 复制链接分享
  • weixin_39830688 weixin_39830688 4月前

    This is likely caused by https://github.com/elastic/logstash/commit/dff953d881a0bf6651ae632751fd506ec7bec1d0#diff-ae70fbcbe615ab91d29724bf87b1c819 added in https://github.com/elastic/logstash/pull/8161 in 6.2.0 and further releases.

    CC

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    but is the redshift jar nowhere to be found now (not even in the folder it was in before 6.2)?

    点赞 评论 复制链接分享
  • weixin_39830688 weixin_39830688 4月前

    It seems to be the case.

    点赞 评论 复制链接分享
  • weixin_39830688 weixin_39830688 4月前

    -brownbear so this really seems to be an issue within Logstash Core rather than this specific input, and it could be affecting more than this. Would you like me to open an issue in Logstash Core? Or given you understand the change/details you prefer to do that?

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    I am trying my best to reproduce this , but coming up short.

    -dudhane19 - a couple questions:

    • How are you installing Logstash (tar, zip, rpm, deb, docker ?)
    • What OS are you running Logstash on ?
    • Is there anything that may make your installations different then just downloading and running ? (custom builds, offline pack, etc. )
    点赞 评论 复制链接分享
  • weixin_39927683 weixin_39927683 4月前

    1) Rpm from elasticsearch repo 2) Centos 7, 64bit 3) add repo, install openjdk, install logstash, install jmx plugin, edit logstash configuration (name, ports, statistics), create one pipeline as described in the first post, run using systemctl

    点赞 评论 复制链接分享
  • weixin_39823459 weixin_39823459 4月前

    Absolutely the same problem with Redshift driver as well

    点赞 评论 复制链接分享
  • weixin_39886547 weixin_39886547 4月前

    Same problem with FileMaker driver

    点赞 评论 复制链接分享
  • weixin_39531582 weixin_39531582 4月前

    Hello,

    I had the similar issue with Logstash 6.2.2. I tried to connect to Redshift database using jdbc driver and got the same error as above.

    Logstash is trying to locate the driver library (i.e. redshift.jar file) in it's classpath - "/logstash-home-dir/logstash-core/lib/jars/". Hence when I copied my redshift.jar file to the classpath, it worked fine.

    Thanks, Sneha

    点赞 评论 复制链接分享
  • weixin_39773817 weixin_39773817 4月前

    This may be related to the Java entry point, which should be a silent feature (e.g. no one should notice it). https://github.com/elastic/logstash/pull/8161.

    It changed the way some of the resources are loaded starting with 6.2.0

    cc: -brownbear

    点赞 评论 复制链接分享
  • weixin_39981093 weixin_39981093 4月前

    hmm but the resource here is the SQL driver jar? That is loaded by the plugin code right? I don't think the changes made to loading the jars are likely to cause this, but maybe I'm missing something here.

    点赞 评论 复制链接分享

相关推荐