View Issue Details

IDProjectCategoryView StatusLast Update
0004332Kali LinuxKali Package Bugpublic2017-11-17 09:20
Reporterdookie Assigned Torhertzog  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Product Version2017.3 
Summary0004332: msfdb init Fails in Live Boot Mode
Description

In live boot mode, msfdb init (and msfdb reinit) fail with the following:

root@kali:~# systemctl start postgresql
root@kali:~# ss -ant
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:5432 :
LISTEN 0 128 ::1:5432 :::*
root@kali:~# msfdb init
Creating database user 'msf'
Enter password for new role:
Enter it again:
Creating databases 'msf' and 'msf_test'
createdb: database creation failed: ERROR: checkpoint request failed
HINT: Consult recent messages in the server log for details.
createdb: database creation failed: ERROR: checkpoint request failed
HINT: Consult recent messages in the server log for details.
Creating configuration file in /usr/share/metasploit-framework/config/database.yml
Creating initial database schema
rake aborted!
ActiveRecord::NoDatabaseError: FATAL: database "msf" does not exist
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:661:in rescue in connect' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:651:inconnect'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:242:in initialize' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:44:innew'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in postgresql_connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:innew_connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in checkout_new_connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:inacquire_connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in block in checkout' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:incheckout'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in block in connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:inconnection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in retrieve_connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_handling.rb:113:inretrieve_connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_handling.rb:87:in connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:941:ininitialize'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:823:in new' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:823:inup'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:801:in migrate' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/tasks/database_tasks.rb:139:inmigrate'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/railties/databases.rake:44:in block (2 levels) in &lt;top (required)>' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rake-12.1.0/exe/rake:27:in<top (required)>'
PG::ConnectionBad: FATAL: database "msf" does not exist
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in initialize' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:651:innew'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in connect' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:242:ininitialize'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in new' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:44:inpostgresql_connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in new_connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:incheckout_new_connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in acquire_connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:inblock in checkout'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in checkout' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:inblock in connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:inretrieve_connection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_handling.rb:113:in retrieve_connection' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_handling.rb:87:inconnection'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:941:in initialize' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:823:innew'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:823:in up' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:801:inmigrate'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/tasks/database_tasks.rb:139:in migrate' /usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/railties/databases.rake:44:inblock (2 levels) in <top (required)>'
/usr/share/metasploit-framework/vendor/bundle/ruby/2.3.0/gems/rake-12.1.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Activities

rhertzog

rhertzog

2017-11-03 13:37

administrator   ~0007575

And the PostgreSQL log says this:
ERROR: could not fsync file "pg_commit_ts": Invalid argument

This really smells like a regression in the kernel, possibly in the overlayfs module. I don't think that the version of Metasploit has anything to do. The fact that it only happens in live mode tends to confirm that it's likely related to the use of overlayfs.

rhertzog

rhertzog

2017-11-07 10:20

administrator   ~0007584

I have sent a report to upstream developers of overlayfs:
https://marc.info/?l=linux-unionfs&amp;m=150971814807948&amp;w=2

I got a query for more data:
https://marc.info/?l=linux-unionfs&amp;m=150978963223298&amp;w=2

And I responded with the required data:
https://marc.info/?l=linux-unionfs&amp;m=151004949511670&amp;w=2

rhertzog

rhertzog

2017-11-07 13:59

administrator   ~0007586

In fact, this seems to be caused by the switch to PostgreSQL 10. The database server tries to fsync() a "pg_commit_ts" directory that has not been modified since the initial installation and this fails because overlayfs delegates the fsync() to the squashfs read-only filesystem that does not have this operation.

I reported this to PostgreSQL developers:
https://www.postgresql.org/message-id/20171107135454.lbelbbvfgadljmuj%40home.ouaza.com

rhertzog

rhertzog

2017-11-07 14:28

administrator   ~0007587

Now that the problem is understood I have added a work-around in kali-defaults 2017.3.2 where we touch a file in /var/lib/postgresql/10/main/pg_commit_ts/ and immediately drop it before starting postgresql.

rhertzog

rhertzog

2017-11-17 09:20

administrator   ~0007608

We have a fix in the kernel as well. So this issue can be closed. I dropped the work-around in git.

Issue History

Date Modified Username Field Change
2017-11-02 16:41 dookie New Issue
2017-11-03 13:37 rhertzog Note Added: 0007575
2017-11-03 13:37 rhertzog Assigned To => rhertzog
2017-11-03 13:37 rhertzog Status new => assigned
2017-11-07 10:20 rhertzog Note Added: 0007584
2017-11-07 13:59 rhertzog Note Added: 0007586
2017-11-07 14:28 rhertzog Note Added: 0007587
2017-11-17 09:20 rhertzog Status assigned => resolved
2017-11-17 09:20 rhertzog Resolution open => fixed
2017-11-17 09:20 rhertzog Note Added: 0007608