Graylog2 Server Currently Unavailable Port 12900



  • from the localhost can you see /system/sessions as accessible?



  • Found the issue. Misdocumentation on the Graylog site, big surprise. Same kinds of issues as before. ElasticSearch was not responding locally and it can't talk to the cluster until the local client is up and running. Got that working and we have Graylog2 up and running!



  • HTTPS vs HTTP?

    Is that port blocked in your FW?



  • @gjacobse said in Graylog2:

    HTTPS vs HTTP?

    Is that port blocked in your FW?

    It's unblocked and tested. It's all HTTP. No proxy in place yet.



  • @DustinB3403 said in Graylog2:

    from the localhost can you see /system/sessions as accessible?

    It doesn't like local...

    curl localhost:9000/system/sessions
    <!DOCTYPE html>
    <html>
      <head>
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="robots" content="noindex, nofollow">
        <meta charset="UTF-8">
        <title>Graylog Web Interface</title>
        <link rel="shortcut icon" href="/favicon.png">
        
      </head>
      <body>
        <script src="/config.js"></script>
        <script src="/vendor.js"></script>
        
        <script src="/polyfill.f2a88026f0d0e7332f17.js"></script>
        
        <script src="/plugin/org.graylog.plugins.pipelineprocessor.ProcessorPlugin/plugin.org.graylog.plugins.pipelineprocessor.PipelineProcessorPlugin.124facd49c87779f691b.js"></script>
        
        <script src="/plugin/org.graylog.plugins.map.MapWidgetPlugin/plugin.org.graylog.plugins.map.MapWidgetPlugin.3a6e86e960b80407f74e.js"></script>
        
        <script src="/plugin/org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin/plugin.org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin.7f1e11d52a66d206799c.js"></script>
        
        <script src="/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.74e8dfee2c1d6b11bc87.js"></script>
        
        <script src="/app.f2a88026f0d0e7332f17.js"></script>
        
      </body>
    </html>
    


  • I've totally disabled the firewall, just to be sure that that is not the problem.



  • Just realized that 12900 is exclusively listening on the loopback.



  • But that doesn't look like it's error'd out, what if you use the IP address rather than localhost.



  • @DustinB3403 said in Graylog2:

    But that doesn't look like it's error'd out, what if you use the IP address rather than localhost.

    It errored out on browser detection.



  • Here is the current status of the configuration file:

    grep -v ^# /etc/graylog/server/server.conf 
    
    is_master = true
    node_id_file = /etc/graylog/server/node-id
    password_secret = [redacted]
    root_password_sha2 = [redacted]
    root_email = [redacted]
    root_timezone = UTC
    plugin_dir = /usr/share/graylog-server/plugin
    rest_listen_uri = http://127.0.0.1:12900/
    web_listen_uri = http://0.0.0.0:9000/
    rotation_strategy = count
    elasticsearch_max_docs_per_index = 20000000
    rotation_strategy = count
    elasticsearch_max_docs_per_index = 20000000
    elasticsearch_max_number_of_indices = 20
    retention_strategy = delete
    elasticsearch_max_number_of_indices = 20
    retention_strategy = delete
    elasticsearch_shards = 4
    elasticsearch_replicas = 0
    elasticsearch_index_prefix = graylog
    allow_leading_wildcard_searches = false
    allow_highlighting = false
    elasticsearch_cluster_name = ntg-graylog2
    elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300, 192.168.1.51:9300, 192.168.1.52:9300, 192.168.1.53:9300, 192.168.1.51:9500
    elasticsearch_node_master = false
    elasticsearch_node_data = false
    elasticsearch_analyzer = standard
    output_batch_size = 500
    output_flush_interval = 1
    output_fault_count_threshold = 5
    output_fault_penalty_seconds = 30
    processbuffer_processors = 5
    outputbuffer_processors = 3
    processor_wait_strategy = blocking
    ring_size = 65536
    inputbuffer_ring_size = 65536
    inputbuffer_processors = 2
    inputbuffer_wait_strategy = blocking
    message_journal_enabled = true
    message_journal_dir = /var/lib/graylog-server/journal
    lb_recognition_period_seconds = 3
    mongodb_uri = mongodb://localhost/graylog
    mongodb_max_connections = 1000
    mongodb_threads_allowed_to_block_multiplier = 5
    content_packs_dir = /usr/share/graylog-server/contentpacks
    




  • Here is the log output on a restart. Getting errors trying to connect to a local service, that I know is running.

    2016-08-20T17:19:14.300Z INFO  [node] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] started
    2016-08-20T17:19:14.547Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:16.789Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:9000]
    2016-08-20T17:19:16.791Z INFO  [HttpServer] [HttpServer] Started.
    2016-08-20T17:19:16.791Z INFO  [WebInterfaceService] Started Web Interface at <http://0.0.0.0:9000/>
    2016-08-20T17:19:17.596Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:19.310Z WARN  [IndexerSetupService] Could not connect to Elasticsearch
    2016-08-20T17:19:19.310Z INFO  [IndexerSetupService] If you're using multicast, check that it is working in your network and that Elasticsearch is accessible. Also check that the cluster name setting is correct.
    2016-08-20T17:19:19.310Z INFO  [IndexerSetupService] See http://docs.graylog.org/en/2.0/pages/configuration/elasticsearch.html for details.
    2016-08-20T17:19:20.648Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:22.266Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:12900]
    2016-08-20T17:19:22.266Z INFO  [HttpServer] [HttpServer-1] Started.
    2016-08-20T17:19:22.266Z INFO  [RestApiService] Started REST API at <http://0.0.0.0:12900/>
    2016-08-20T17:19:22.267Z INFO  [ServiceManagerListener] Services are healthy
    2016-08-20T17:19:22.268Z INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=196, BufferSynchronizerService [RUNNING]=202, MetricsReporterService [RUNNING]=203, KafkaJournal [RUNNING]=240, InputSetupService [RUNNING]=276, PeriodicalsService [RUNNING]=642, JournalReader [RUNNING]=854, WebInterfaceService [RUNNING]=6864, IndexerSetupService [RUNNING]=9386, RestApiService [RUNNING]=12343}
    2016-08-20T17:19:22.272Z INFO  [ServerBootstrap] Graylog server up and running.
    2016-08-20T17:19:22.277Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
    2016-08-20T17:19:23.711Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:25.121Z INFO  [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
    2016-08-20T17:19:26.736Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:29.764Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:32.785Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:35.810Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:38.826Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:41.845Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:44.860Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:47.874Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    2016-08-20T17:19:50.889Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
    


  • I made the same change as the person in the GitHub thread and opened port 12900 to the world so that I could test from a browser. And it indeed can be reached for me, the same as him. So the error appears to be wrong.

    0_1471713807470_Screenshot from 2016-08-20 13-23-18.png



  • Found the issue. Misdocumentation on the Graylog site, big surprise. Same kinds of issues as before. ElasticSearch was not responding locally and it can't talk to the cluster until the local client is up and running. Got that working and we have Graylog2 up and running!



  • Don't forget to mark the thread as solved then 😉



  • @StrongBad said in Graylog2:

    Don't forget to mark the thread as solved then 😉

    Marked


Log in to reply