You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When we restarted the GoCD server as part of regular patching of our underlying EKS worker nodes, there were some problems migrating the h2 database, hence preventing GoCD server from starting up properly. In order to mitigate this db migration error, we had to manually export and import the db data by using the steps mentioned in the below link
Browser vendor and version (if relevant): Chrome/Firefox
Expected Results
GoCD server should restart successfully
Actual Results
ERROR [WrapperJarAppMain] DatabaseMigrator:65 - Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
Possible Fix
Log snippets
"Running go-server..."
wrapper | Failed to set JVM input handle to non blocking mode: Bad file descriptor (9)
wrapper | Failed to set JVM input handle to close on JVM exit: Bad file descriptor (9)
"wrapper | JVM exited while loading the application."
"wrapper | Unable to set JVM's stdin: Bad file descriptor"
"wrapper | --> Wrapper Started as Console"
�]0;go-server�wrapper | Java Service Wrapper Standard Edition 64-bit 3.5.50
wrapper | Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
"wrapper | http://wrapper.tanukisoftware.com"
"wrapper | Licensed to ThoughtWorks for GoCD Server"
"wrapper | "
"wrapper | Launching a JVM..."
jvm 1 | wrapper | --> Wrapper Started as Console
jvm 1 | �]0;go-server�wrapper | Java Service Wrapper Standard Edition 64-bit 3.5.50
jvm 1 | wrapper | Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
jvm 1 | wrapper | http://wrapper.tanukisoftware.com
jvm 1 | wrapper | Licensed to ThoughtWorks for GoCD Server
jvm 1 | wrapper |
jvm 1 | wrapper | Launching a JVM...
"wrapper | Launching a JVM..."
jvm 2 | -XX:CompressedClassSpaceSize=335544320 -XX:ConcGCThreads=2 -XX:G1ConcRefinementThreads=8 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=2147483648 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=4294967296 -XX:MaxMetaspaceSize=419430400 -XX:MinHeapSize=2147483648 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC
jvm 2 | WrapperManager: Initializing...
jvm 2 | [Fri Jan 27 14:42:36 GMT 2023] Starting process:
jvm 2 | [Fri Jan 27 14:42:36 GMT 2023] Working directory : /go-working-dir
jvm 2 | [Fri Jan 27 14:42:36 GMT 2023] Application arguments: []
jvm 2 | [Fri Jan 27 14:42:36 GMT 2023] GoCD Version: 22.3.0-15301
jvm 2 | [Fri Jan 27 14:42:36 GMT 2023] Java Version: 17.0.5
jvm 2 | [Fri Jan 27 14:42:36 GMT 2023] Operating System: Linux(5.4.228-131.415.amzn2.x86_64)
jvm 2 | Could not find file `config/logback.xml'. Attempting to load from classpath.
jvm 2 | Using classpath resource `jar:onejar:lib/server-launcher-22.3.0-15301-main.jar!/config/logback.xml'.
jvm 2 | 2023-01-27 14:42:37,413 INFO [WrapperJarAppMain] Jetty9Server:193 - Configuring Jetty using /go-working-dir/config/jetty.xml
jvm 2 | 2023-01-27 14:42:37,469 WARN [WrapperJarAppMain] Server:357 - ErrorPageMapper not supported for Server level Error Handling
jvm 2 | 2023-01-27 14:42:37,539 WARN [WrapperJarAppMain] AbstractHandler:96 - No Server set for ResourceHandler@54d62683{STOPPED}
jvm 2 | 2023-01-27 14:42:43,244 WARN [WrapperJarAppMain] ConnectionManager:117 - The file config/db.properties specified by `go.db.config` does not exist.
jvm 2 | 2023-01-27 14:42:45,050 INFO [WrapperJarAppMain] DatabaseMigrator:40 - Upgrading database, this might take a while depending on the size of the database.
jvm 2 | ************************************************************************
jvm 2 | 2023-01-27 14:42:45,051 INFO [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
jvm 2 | WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
jvm 2 | 2023-01-27 14:42:45,051 INFO [WrapperJarAppMain] DatabaseMigrator:49 - WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
jvm 2 | ************************************************************************
jvm 2 | 2023-01-27 14:42:45,051 INFO [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
jvm 2 | Jan 27, 2023 2:42:45 PM liquibase.database
jvm 2 | INFO: Set default schema name to PUBLIC
jvm 2 | Jan 27, 2023 2:42:45 PM liquibase.lockservice
jvm 2 | INFO: Waiting for changelog lock....
jvm 2 | 2023-01-27 14:42:55,176 ERROR [WrapperJarAppMain] DatabaseMigrator:65 - Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]. The cause was: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | liquibase.exception.LockException: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:331)
jvm 2 | at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:249)
jvm 2 | at liquibase.Liquibase.lambda$update$1(Liquibase.java:239)
jvm 2 | at liquibase.Scope.lambda$child$0(Scope.java:180)
jvm 2 | at liquibase.Scope.child(Scope.java:189)
jvm 2 | at liquibase.Scope.child(Scope.java:179)
jvm 2 | at liquibase.Scope.child(Scope.java:158)
jvm 2 | at liquibase.Liquibase.runInScope(Liquibase.java:2447)
jvm 2 | at liquibase.Liquibase.update(Liquibase.java:236)
jvm 2 | at liquibase.Liquibase.update(Liquibase.java:221)
jvm 2 | at liquibase.Liquibase.update(Liquibase.java:207)
jvm 2 | at com.thoughtworks.go.server.database.migration.DatabaseMigrator.migrate(DatabaseMigrator.java:54)
jvm 2 | at com.thoughtworks.go.server.database.Database.getDataSource(Database.java:73)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2 | at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1176)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1071)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
jvm 2 | at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:211)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1134)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:927)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:804)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:573)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
jvm 2 | at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:206)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1263)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1097)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
jvm 2 | at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
jvm 2 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
jvm 2 | at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
jvm 2 | at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
jvm 2 | at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
jvm 2 | at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
jvm 2 | at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
jvm 2 | at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
jvm 2 | at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
jvm 2 | at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
jvm 2 | at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
jvm 2 | at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
jvm 2 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 2 | at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
jvm 2 | at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
jvm 2 | at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:517)
jvm 2 | at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:157)
jvm 2 | at com.thoughtworks.go.server.Jetty9Server.startHandlers(Jetty9Server.java:173)
jvm 2 | at liquibase.Liquibase.runInScope(Liquibase.java:2447)
jvm 2 | at liquibase.Liquibase.update(Liquibase.java:236)
jvm 2 | at liquibase.Liquibase.update(Liquibase.java:221)
jvm 2 | at liquibase.Liquibase.update(Liquibase.java:207)
jvm 2 | at com.thoughtworks.go.server.database.migration.DatabaseMigrator.migrate(DatabaseMigrator.java:54)
jvm 2 | at com.thoughtworks.go.server.database.Database.getDataSource(Database.java:73)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2 | at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1176)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1071)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
jvm 2 | at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:211)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1134)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:927)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:804)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:573)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
jvm 2 | at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:206)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1263)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1097)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2 | at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
jvm 2 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
jvm 2 | at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
jvm 2 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
jvm 2 | at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
jvm 2 | at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
jvm 2 | at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
jvm 2 | at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
jvm 2 | at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
jvm 2 | at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
jvm 2 | at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
jvm 2 | at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
jvm 2 | at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
jvm 2 | at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
jvm 2 | at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
jvm 2 | at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 2 | at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
jvm 2 | at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
jvm 2 | at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:517)
jvm 2 | at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:157)
jvm 2 | at com.thoughtworks.go.server.Jetty9Server.startHandlers(Jetty9Server.java:173)
jvm 2 | at com.thoughtworks.go.server.Jetty9Server.start(Jetty9Server.java:129)
jvm 2 | at com.thoughtworks.go.server.GoServer.startServer(GoServer.java:62)
jvm 2 | at com.thoughtworks.go.server.GoServer.go(GoServer.java:54)
jvm 2 | at com.thoughtworks.go.server.util.GoLauncher.main(GoLauncher.java:42)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2 | at com.thoughtworks.gocd.Boot.run(Boot.java:90)
jvm 2 | at com.thoughtworks.gocd.Boot.main(Boot.java:56)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2 | at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
jvm 2 | at java.base/java.lang.Thread.run(Unknown Source)
jvm 2 | Caused by: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | at liquibase.database.jvm.JdbcConnection.rollback(JdbcConnection.java:489)
jvm 2 | at liquibase.database.AbstractJdbcDatabase.rollback(AbstractJdbcDatabase.java:1172)
jvm 2 | at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:287)
jvm 2 | ... 92 common frames omitted
jvm 2 | Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | at org.h2.message.DbException.getJdbcSQLException(DbException.java:505)
jvm 2 | at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
jvm 2 | at org.h2.message.DbException.get(DbException.java:194)
jvm 2 | at org.h2.engine.Database.throwLastBackgroundException(Database.java:2221)
jvm 2 | at org.h2.engine.Database.commit(Database.java:2202)
jvm 2 | at org.h2.engine.Session.rollback(Session.java:846)
jvm 2 | at org.h2.command.dml.TransactionCommand.update(TransactionCommand.java:49)
jvm 2 | at org.h2.command.CommandContainer.update(CommandContainer.java:198)
jvm 2 | at org.h2.command.Command.executeUpdate(Command.java:251)
jvm 2 | at org.h2.jdbc.JdbcConnection.rollbackInternal(JdbcConnection.java:1530)
jvm 2 | at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:561)
jvm 2 | at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:781)
jvm 2 | at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:781)
jvm 2 | at liquibase.database.jvm.JdbcConnection.rollback(JdbcConnection.java:486)
jvm 2 | ... 94 common frames omitted
jvm 2 | Caused by: org.h2.message.DbException: General error: "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0" [50000-200]
jvm 2 | at org.h2.message.DbException.get(DbException.java:194)
jvm 2 | at org.h2.message.DbException.convert(DbException.java:347)
jvm 2 | at org.h2.mvstore.db.MVTableEngine$1.uncaughtException(MVTableEngine.java:93)
jvm 2 | at org.h2.mvstore.MVStore.handleException(MVStore.java:2877)
jvm 2 | at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2813)
jvm 2 | at org.h2.mvstore.MVStore$BackgroundWriterThread.run(MVStore.java:3290)
jvm 2 | Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0" [50000-200]
jvm 2 | at org.h2.message.DbException.getJdbcSQLException(DbException.java:505)
jvm 2 | at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
jvm 2 | ... 6 common frames omitted
jvm 2 | Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
jvm 2 | at org.h2.mvstore.Page.getKey(Page.java:270)
jvm 2 | at org.h2.mvstore.MVMap.rewritePage(MVMap.java:773)
jvm 2 | at org.h2.mvstore.MVMap.rewrite(MVMap.java:734)
jvm 2 | at org.h2.mvstore.MVMap.rewrite(MVMap.java:710)
jvm 2 | at org.h2.mvstore.MVStore.compactRewrite(MVStore.java:2137)
jvm 2 | at org.h2.mvstore.MVStore.rewriteChunks(MVStore.java:2026)
jvm 2 | at org.h2.mvstore.MVStore.doMaintenance(MVStore.java:2844)
jvm 2 | at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2788)
jvm 2 | ... 1 common frames omitted
jvm 2 | Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]. The cause was: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | liquibase.exception.LockException: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:331)
jvm 2 | at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:249)
jvm 2 | at liquibase.Liquibase.lambda$update$1(Liquibase.java:239)
jvm 2 | at liquibase.Scope.lambda$child$0(Scope.java:180)
jvm 2 | at liquibase.Scope.child(Scope.java:189)
jvm 2 | at com.thoughtworks.go.server.Jetty9Server.start(Jetty9Server.java:129)
jvm 2 | at com.thoughtworks.go.server.GoServer.startServer(GoServer.java:62)
jvm 2 | at com.thoughtworks.go.server.GoServer.go(GoServer.java:54)
jvm 2 | at com.thoughtworks.go.server.util.GoLauncher.main(GoLauncher.java:42)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2 | at com.thoughtworks.gocd.Boot.run(Boot.java:90)
jvm 2 | at com.thoughtworks.gocd.Boot.main(Boot.java:56)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2 | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2 | at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
jvm 2 | at java.base/java.lang.Thread.run(Unknown Source)
jvm 2 | Caused by: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
jvm 2 | at liquibase.database.jvm.JdbcConnection.rollback(JdbcConnection.java:489)
jvm 2 | at liquibase.database.AbstractJdbcDatabase.rollback(AbstractJdbcDatabase.java:1172)
jvm 2 | at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:287)
jvm 2 | ... 92 common frames omitted
jvm 2 | Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2 | ROLLBACK [50000-200]
The text was updated successfully, but these errors were encountered:
Unfortunately, this points to H2 database corruption (which you have worked around with the export+import). If your H2 database has been corrupted somehow, there's not a lot GoCD can do to recover it automatically.
So the key here is to figure out what caused the corruption in the first place and whether there's anything GoCD can do about it. I would check
was the server otherwise working OK and had been restarted (on the earlier version) with no similar h2 stack traces or start-up errors earlier?
were there similar h2 stack traces in the logs for, say, a week or two prior?
if not, during the upgrade, did you have problems or did anything happen such as the pod being manually deleted while start, or Kubernetes or the Tanuki wrapper auto-restarting the JVM for some reason?
This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.
Issue Type
Summary
When we restarted the GoCD server as part of regular patching of our underlying EKS worker nodes, there were some problems migrating the h2 database, hence preventing GoCD server from starting up properly. In order to mitigate this db migration error, we had to manually export and import the db data by using the steps mentioned in the below link
https://gist.github.com/arvindsv/c7934bee26ce1c47e9d24f0a493d3c87
Environment
Basic environment details
GoCD Version: 22.3.0-15301
17.0.5
Linux(5.4.228-131.415.amzn2.x86_64)
Chrome/Firefox
Expected Results
GoCD server should restart successfully
Actual Results
ERROR [WrapperJarAppMain] DatabaseMigrator:65 - Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
Possible Fix
Log snippets
The text was updated successfully, but these errors were encountered: