Unable to Uninstall & Re-install MemSQL 7.1 Nodes: "Failed to locate any of the following files: /var/lib/memsql/nodes.hcl"

[ymonye]

Hi,

I’ve experienced a hard drive failure and now attempting to re-install MemSQL under the “Self-Managed” steps, on Debian Linux, as previously done prior to the failure.

Linux · SingleStore Documentation

I’m able to setup a new host memsql-toolbox-config register-host --localhost --host 127.0.0.1 however, I see the following error when attempting to create a node under that host:

#memsql-admin create-node --host 127.0.0.1 --password “” -P 3307 --base-install-dir /misc/memsql/master

could not determine memsqlctl user: error running memsqlctl: error running command: "/usr/bin/memsqlctl" "--json" "--yes" "env": exit status 1**
stderr: Failed to locate any of the following files: /var/lib/memsql/nodes.hcl
See memsqlctl · SingleStore Documentation for more information

That above link points to a completely different & unrelated article found here. Perhaps that should be updated in future MemSQL versions?:

https://docs.memsql.com/v7.1/reference/memsql-tools-reference/memsqlctl-reference/memsqlctl-update-config/

I’ve attempted to uninstall memsql, then reinstall, yet still receive the same error. Since the /var/lib/memsql/ path was completely erased, is there any way to get memsql to reinstall it? Or even better, is there any procedure to have a clean uninstallation of memsql, so that new config files are written upon a re-install?

Thanks

[ymonye]

I couldn’t find any clean method of uninstalling MemSQL 7.1 from this Linux machine, and the /var/lib/memsql path with the previous node configuration remained even after uninstalling memsql-server 7.1, memsql-client, memsql-toolbox, & memsql-studio.

I went ahead and removed all versions (including previous versions) of memsql-server by identifying them with memsql-deploy install --all, reinstalled memsql-client, memsql-toolbox, memsql-studio, registered a new localhost, then reinstalled memsql-server 7.1, yet I’m still unable to add nodes.

Here’s what I see after attempting to create a node.

#memsql-admin create-node --host 127.0.0.1 --password “” -P 3307 --base-install-dir /misc/memsql/master
user@127.0.0.1’s password:
The target user user on host 127.0.0.1 does not have the privileges to perform this action.
Please enter your password to proceed with sudo. (For details, see SingleStore Tools Issues · SingleStore Documentation).
sudo password for user@127.0.0.1:
Toolbox is about to perform the following actions on host 127.0.0.1:
· Run ‘memsqlctl create-node --port 3307 --base-install-dir /misc/memsql/master --password ●●●●●●’

Would you like to continue? [y/N]: y
error running memsqlctl: error running command: "/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/run/user/1000/memsql-toolbox022871596/1.socket" "127.0.0.1" "'sudo'" "'-S'" "'-k'" "'--'" "'/bin/sh'" "'-c'" "''\\''printf'\\'' '\\''20bd5aa4-4b69-42ef-bd0e-d0cba3961d0c'\\'' && '\\''exec'\\'' '\\''memsqlctl'\\'' '\\''--json'\\'' '\\''--yes'\\'' '\\''create-node'\\'' '\\''--base-install-dir'\\'' '\\''/misc/memsql/master'\\'' '\\''--port'\\'' '\\''3307'\\'' '\\''--secure-password'\\'' '\\''7uDmHbA1V5UWtmVVV/vPpZMCWtvo8C/WOb+BYQ=='\\'''": exit status 1
stderr: Latest errors from MemSQL tracelog:
48 2020-06-12 23:56:50.234 INFO: Log opened
: Failed to connect to MemSQL: Could not establish connection: Error 1045: Access denied for user ‘root’@‘localhost’ (using password: NO)

For some reason even if the above fails, MemSQL still decides to add that node information to the /var/lib/memsql/nodes.hcl file. This is made apparent to me after attempting to re-run the previous command, resulting in the below:

#memsql-admin create-node --host 127.0.0.1 --password “” -P 3307 --base-install-dir /misc/memsql/master
user@127.0.0.1’s password:
The target user user on host 127.0.0.1 does not have the privileges to perform this action.
Please enter your password to proceed with sudo. (For details, see SingleStore Tools Issues · SingleStore Documentation).
sudo password for user@127.0.0.1:
Toolbox is about to perform the following actions on host 127.0.0.1:
· Run ‘memsqlctl create-node --port 3307 --base-install-dir /misc/memsql/master --password ●●●●●●’

Would you like to continue? [y/N]: y
error running memsqlctl: error running command: "/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/run/user/1000/memsql-toolbox069168063/1.socket" "127.0.0.1" "'sudo'" "'-S'" "'-k'" "'--'" "'/bin/sh'" "'-c'" "''\\''printf'\\'' '\\''ab5bb876-1ab7-4e7f-b0dc-9ee3d5459e19'\\'' && '\\''exec'\\'' '\\''memsqlctl'\\'' '\\''--json'\\'' '\\''--yes'\\'' '\\''create-node'\\'' '\\''--base-install-dir'\\'' '\\''/misc/memsql/master'\\'' '\\''--port'\\'' '\\''3307'\\'' '\\''--secure-password'\\'' '\\''O7BXaPOpldLPVp0FPhN/tEWuspRlQeVc5DtTYw=='\\'''": exit status 1
stderr: memsqlctl expects to create a new file for the MemSQL config at /misc/memsql/master/memsql.cnf, but a file already exists at this path. Run ‘memsqlctl register-node’ if you intend to register an existing node with the provided config.

Attempting to view any existing nodes produces the below:

#memsql-admin list-nodes
user@127.0.0.1’s password:
✘ Failed to list nodes on all hosts: failed to list nodes on host 127.0.0.1: error running memsqlctl: error running command: "/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/run/user/1000/memsql-toolbox685300850/1.socket" "127.0.0.1" "'memsqlctl'" "'--json'" "'--yes'" "'list-nodes'": exit status 1
stderr: memsqlctl failed to load the MemSQL config file ‘/misc/memsql/master/memsql.cnf’ referenced by the node metadata file at ‘/var/lib/memsql/nodes.hcl’.
See update-config · SingleStore Documentation for more information.
No nodes found

[vtkachuk-ua]

Hello, @ymonye.

The error messages look a bit wierd. Could you help me with the investigation?
first of all, could you install the latest toolbox (1.5.5), and then proceed with the following steps:

1. To uninstall existing installation:
   $ memsql-admin delete-node --all --stop -vvv
   $ memsql-deploy uninstall --all-versions -vvv
   $ memsql-toolbox-config unregister-host --all -vvv

2. To re-install:
   $ memsql-toolbox-config register-host --localhost --host 127.0.0.1 -vvv
   $ memsql-deploy install --all --version 7.1 -vvv
   $ memsql-admin create-node --host 127.0.0.1 --password “” -P 3307 --base-install-dir /misc/memsql/master -vvv

Could you please share log output from commands?

[ymonye]

@vtkachuk-ua, here’s what I have. In summary, I ran into issues at the very last step of creating a new node, using the same port & base install directory as before, despite the fact the node & directory at /misc/memsql directory were removed prior to running each command.

I went ahead and created another node on a different port (3317) and directory, and that was successful. However, I received another error after attempting to verify the node was created successfully, after running: memsql-admin list-nodes.

After checking the /var/lib/memsql/nodes.hcl file, it shows that two nodes were created, including the failed node on port 3307. All logs are provided in the post below.

RESULTS:

memsql-admin delete-node --all --stop -vvv

2020/06/16 16:23:22 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:23:22 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:23:22 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox205677452 SSHBinaryPath:/usr/bin/ssh}
2020/06/16 16:23:22 logging.go:37 Running command #1 /usr/bin/id -u
2020/06/16 16:23:22 logging.go:52 Command #1 took 876.919µs
2020/06/16 16:23:22 logging.go:37 Running command #2 /usr/bin/which which
2020/06/16 16:23:22 logging.go:52 Command #2 took 846.148µs
2020/06/16 16:23:22 logging.go:37 Running command #3 /usr/bin/which memsqlctl
2020/06/16 16:23:22 logging.go:52 Command #3 took 683.567µs
2020/06/16 16:23:22 logging.go:37 Running command #4 /usr/bin/memsqlctl --json --yes env
2020/06/16 16:23:22 logging.go:52 Command #4 took 6.945539ms
2020/06/16 16:23:22 logging.go:37 Running command #5 /usr/bin/id -G
2020/06/16 16:23:22 logging.go:52 Command #5 took 831.509µs
2020/06/16 16:23:22 logging.go:37 Running command #6 /usr/bin/memsqlctl --json --yes list-nodes
2020/06/16 16:23:22 logging.go:52 Command #6 took 4.746751ms
There are no nodes to delete

memsql-deploy uninstall --all-versions -vvv

2020/06/16 16:24:27 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:24:27 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:24:27 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox133047590 SSHBinaryPath:/usr/bin/ssh}
2020/06/16 16:24:27 logging.go:37 Running command #1 /usr/bin/which which
2020/06/16 16:24:27 logging.go:52 Command #1 took 764.151µs
2020/06/16 16:24:27 logging.go:37 Running command #2 /usr/bin/which rpm
2020/06/16 16:24:27 logging.go:52 Command #2 failed with exit code 1 in 736.756µs
2020/06/16 16:24:27 logging.go:37 Running command #3 /usr/bin/which which
2020/06/16 16:24:27 logging.go:52 Command #3 took 696.841µs
2020/06/16 16:24:27 logging.go:37 Running command #4 /usr/bin/which dpkg
2020/06/16 16:24:27 logging.go:52 Command #4 took 679.867µs
2020/06/16 16:24:27 logging.go:37 Running command #5 /usr/bin/id -u
2020/06/16 16:24:27 logging.go:52 Command #5 took 785.701µs
2020/06/16 16:24:27 logging.go:37 Running command #6 /usr/bin/which which
2020/06/16 16:24:27 logging.go:52 Command #6 took 678.278µs
2020/06/16 16:24:27 logging.go:37 Running command #7 /usr/bin/which sudo
2020/06/16 16:24:27 logging.go:52 Command #7 took 662.128µs
2020/06/16 16:24:27 logging.go:37 Running command #8 /usr/bin/sudo -n -k id
2020/06/16 16:24:27 logging.go:52 Command #8 failed with exit code 1 in 3.61877ms
2020/06/16 16:24:27 logging.go:37 Running command #9 /usr/bin/id -u -n
2020/06/16 16:24:27 logging.go:52 Command #9 took 1.082222ms
The target user user on host 127.0.0.1 does not have the privileges to perform this action.
Please enter your password to proceed with sudo. (For details, see SingleStore Tools Issues · SingleStore Documentation).
sudo password for user@127.0.0.1:
2020/06/16 16:24:33 logging.go:37 Running command #10 /usr/bin/timeout --foreground 5 sudo -S -k -v
2020/06/16 16:24:34 logging.go:52 Command #10 took 24.256475ms
2020/06/16 16:24:34 logging.go:37 Running command #11 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec which which
2020/06/16 16:24:34 logging.go:52 Command #11 took 11.920182ms
2020/06/16 16:24:34 logging.go:37 Running command #12 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec which memsqlctl
2020/06/16 16:24:34 logging.go:52 Command #12 took 15.063242ms
2020/06/16 16:24:34 logging.go:37 Running command #13 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec readlink --canonicalize --no-newline /usr/bin/memsqlctl
2020/06/16 16:24:34 logging.go:52 Command #13 took 11.827142ms
2020/06/16 16:24:34 logging.go:37 Running command #14 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec dpkg-query --search /opt/memsql-server-7.1.2-9ba41ab64a/memsqlctl
2020/06/16 16:24:34 logging.go:52 Command #14 took 212.621788ms
Silly 2020/06/16 16:24:34 dpkg.go:173 STDERR of “dpkg-query --search”: [sudo] password for user:
2020/06/16 16:24:34 logging.go:37 Running command #15 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec dpkg-query --show --showformat ${Package} ${Version} ${db:Status-Abbrev}\n
2020/06/16 16:24:34 logging.go:52 Command #15 took 46.577679ms
Silly 2020/06/16 16:24:34 dpkg.go:97 STDERR of “dpkg-query --show”: [sudo] password for user:
2020/06/16 16:24:34 logging.go:37 Running command #16 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec memsqlctl --json --yes list-nodes
2020/06/16 16:24:34 logging.go:52 Command #16 took 16.364605ms
Toolbox will perform the following actions:
· Uninstall all versions of memsql-server on 127.0.0.1

Would you like to continue? [y/N]: y
2020/06/16 16:24:38 logging.go:37 Running command #17 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec which which
2020/06/16 16:24:38 logging.go:52 Command #17 took 11.983592ms
2020/06/16 16:24:38 logging.go:37 Running command #18 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec which memsqlctl
2020/06/16 16:24:38 logging.go:52 Command #18 took 11.993507ms
2020/06/16 16:24:38 logging.go:37 Running command #19 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec readlink --canonicalize --no-newline /usr/bin/memsqlctl
2020/06/16 16:24:38 logging.go:52 Command #19 took 11.525245ms
2020/06/16 16:24:38 logging.go:37 Running command #20 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec dpkg-query --search /opt/memsql-server-7.1.2-9ba41ab64a/memsqlctl
2020/06/16 16:24:38 logging.go:52 Command #20 took 134.887167ms
Silly 2020/06/16 16:24:38 dpkg.go:173 STDERR of “dpkg-query --search”: [sudo] password for user:
2020/06/16 16:24:38 logging.go:37 Running command #21 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec dpkg-query --show --showformat ${Package} ${Version} ${db:Status-Abbrev}\n
2020/06/16 16:24:38 logging.go:52 Command #21 took 50.236065ms
Silly 2020/06/16 16:24:38 dpkg.go:97 STDERR of “dpkg-query --show”: [sudo] password for user:
2020/06/16 16:24:38 logging.go:37 Running command #22 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 79f5d93b-bcdb-45b2-aa46-5c64b40a6208 && exec dpkg --remove memsql-server7.1.2-9ba41ab64a
2020/06/16 16:24:38 logging.go:52 Command #22 took 245.184661ms
Silly 2020/06/16 16:24:38 dpkg.go:138 STDERR of “dpkg --remove”: [sudo] password for user: update-alternatives: warning: alternative /opt/memsql-server-7.1.2-9ba41ab64a/memsqlctl (part of link group memsqlctl) doesn’t exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/memsqlctl is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/memsql-server-7.1.2-9ba41ab64a/memsql_exporter/memsql_exporter (part of link group memsql_exporter) doesn’t exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/memsql_exporter is dangling; it will be updated with best choice
update-alternatives: warning: alternative /opt/memsql-server-7.1.2-9ba41ab64a/memsql_exporter/memsql_pusher (part of link group memsql_pusher) doesn’t exist; removing from list of alternatives
update-alternatives: warning: /etc/alternatives/memsql_pusher is dangling; it will be updated with best choice
✓ Uninstalled on host 127.0.0.1 (1/1)
✓ Successfully uninstalled all versions of memsql-server on 1 host
Operation completed successfully

memsql-toolbox-config unregister-host --all -vvv

2020/06/16 16:25:25 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:25:25 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:25:25 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox420146615 SSHBinaryPath:/usr/bin/ssh}
Toolbox will perform the following actions:
· Unregister host 127.0.0.1

Would you like to continue? [y/N]: y
2020/06/16 16:25:27 logging.go:37 Running command #1 /usr/bin/id -u
2020/06/16 16:25:27 logging.go:52 Command #1 took 931.492µs
2020/06/16 16:25:27 logging.go:37 Running command #2 /usr/bin/which which
2020/06/16 16:25:27 logging.go:52 Command #2 took 698.886µs
2020/06/16 16:25:27 logging.go:37 Running command #3 /usr/bin/which memsqlctl
2020/06/16 16:25:27 logging.go:52 Command #3 failed with exit code 1 in 742.983µs
✓ Successfully unregistered 1 host
If you want to re-register the hosts use the following commands:

memsql-toolbox-config register-host ‘–host’ ‘127.0.0.1’ ‘–localhost’

Operation completed successfully

memsql-toolbox-config register-host --localhost --host 127.0.0.1 -vvv

2020/06/16 16:26:00 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:26:00 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:26:00 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox427819352 SSHBinaryPath:/usr/bin/ssh}
WARNING: Specifying both --localhost and --host 127.0.0.1 is a mistake in most cases unless you want to setup cluster-in-a-box.
You need to use a cluster-addressable IP for this host so that it can be accessed from other hosts in the cluster.
Do you want to continue with your configuration? [y/N]: y
Toolbox is about to register the following host:
· Host: 127.0.0.1
· Localhost: true

Would you like to continue? [y/N]: y
✓ Successfully registered host 127.0.0.1
±----------±-----------±------------±--------------+
| Host | Local Host | SSH address | Identity File |
±----------±-----------±------------±--------------+
| 127.0.0.1 | Yes | | |
±----------±-----------±------------±--------------+

memsql-deploy install --all --version 7.1 -vvv

2020/06/16 16:26:49 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:26:49 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:26:49 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox648910337 SSHBinaryPath:/usr/bin/ssh}
2020/06/16 16:26:49 logging.go:37 Running command #1 /usr/bin/which which
2020/06/16 16:26:49 logging.go:52 Command #1 took 801.678µs
2020/06/16 16:26:49 logging.go:37 Running command #2 /usr/bin/which rpm
2020/06/16 16:26:49 logging.go:52 Command #2 failed with exit code 1 in 827.817µs
2020/06/16 16:26:49 logging.go:37 Running command #3 /usr/bin/which which
2020/06/16 16:26:49 logging.go:52 Command #3 took 614.34µs
2020/06/16 16:26:49 logging.go:37 Running command #4 /usr/bin/which dpkg
2020/06/16 16:26:49 logging.go:52 Command #4 took 672.918µs
2020/06/16 16:26:49 logging.go:37 Running command #5 /usr/bin/id -u
2020/06/16 16:26:49 logging.go:52 Command #5 took 847.635µs
2020/06/16 16:26:49 logging.go:37 Running command #6 /usr/bin/which which
2020/06/16 16:26:49 logging.go:52 Command #6 took 710.12µs
2020/06/16 16:26:49 logging.go:37 Running command #7 /usr/bin/which sudo
2020/06/16 16:26:49 logging.go:52 Command #7 took 674.908µs
2020/06/16 16:26:49 logging.go:37 Running command #8 /usr/bin/sudo -n -k id
2020/06/16 16:26:49 logging.go:52 Command #8 failed with exit code 1 in 3.702506ms
2020/06/16 16:26:49 logging.go:37 Running command #9 /usr/bin/id -u -n
2020/06/16 16:26:49 logging.go:52 Command #9 took 1.089481ms
The target user user on host 127.0.0.1 does not have the privileges to perform this action.
Please enter your password to proceed with sudo. (For details, see SingleStore Tools Issues · SingleStore Documentation).
sudo password for user@127.0.0.1:
2020/06/16 16:26:55 logging.go:37 Running command #10 /usr/bin/timeout --foreground 5 sudo -S -k -v
2020/06/16 16:26:55 logging.go:52 Command #10 took 10.57205ms
Toolbox will perform the following actions:
· Download memsql-server 7.1.3-0925a98cb8 (7.1)

· Install memsql-server 7.1.3-0925a98cb8 (7.1) on 127.0.0.1

Would you like to continue? [y/N]: y
Debug 2020/06/16 16:26:58 downloader.go:91 Fetching package meta from https://release.memsql.com/production/index/memsqlserver/7.1.json
Debug 2020/06/16 16:26:58 downloader.go:42 Downloading from Freya into /tmp/memsql-server-deb260764524
Debug 2020/06/16 16:26:58 downloader.go:129 Downloading https://release.memsql.com/production/debian/pool/memsql-server_7.1.3_0925a98cb8_amd64.deb
✓ Downloaded memsql-server production:7.1
2020/06/16 16:27:03 logging.go:37 Running command #11 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec which which
2020/06/16 16:27:03 logging.go:52 Command #11 took 11.939835ms
2020/06/16 16:27:03 logging.go:37 Running command #12 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec which memsqlctl
2020/06/16 16:27:03 logging.go:52 Command #12 failed with exit code 1 in 11.630096ms
2020/06/16 16:27:03 logging.go:37 Running command #13 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec dpkg-query --show --showformat ${Package} ${Version} ${db:Status-Abbrev}\n
2020/06/16 16:27:03 logging.go:52 Command #13 took 46.799019ms
Silly 2020/06/16 16:27:03 dpkg.go:97 STDERR of “dpkg-query --show”: [sudo] password for user:
2020/06/16 16:27:03 logging.go:37 Running command #14 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec printenv SUDO_USER
2020/06/16 16:27:03 logging.go:52 Command #14 took 12.534677ms
2020/06/16 16:27:03 logging.go:37 Running command #15 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec dpkg --install /tmp/memsql-server-deb260764524
2020/06/16 16:27:06 logging.go:52 Command #15 took 3.343003748s
Silly 2020/06/16 16:27:06 dpkg.go:117 STDERR of “dpkg --install”: [sudo] password for user: Created symlink /etc/systemd/system/multi-user.target.wants/memsql.service → /opt/memsql-server-7.1.3-0925a98cb8/memsql.service.
Created symlink /etc/systemd/system/memsql.service → /opt/memsql-server-7.1.3-0925a98cb8/memsql.service.
2020/06/16 16:27:06 logging.go:37 Running command #16 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec getent group memsql
2020/06/16 16:27:06 logging.go:52 Command #16 took 12.309691ms
2020/06/16 16:27:06 logging.go:37 Running command #17 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 4c432dd7-8d10-454e-8ff0-aa924e65f35f && exec usermod -a -G memsql user
2020/06/16 16:27:06 logging.go:52 Command #17 took 15.243919ms
✓ Installed memsql-server7.1.3-0925a98cb8 on host 127.0.0.1 (1/1)
✓ Successfully installed on 1 host
Operation completed successfully

memsql-admin create-node --host 127.0.0.1 --password "" -P 3307 --base-install-dir /misc/memsql/master -vvv

2020/06/16 16:28:32 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:28:32 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:28:32 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox309711868 SSHBinaryPath:/usr/bin/ssh}
2020/06/16 16:28:32 logging.go:37 Running command #1 /usr/bin/id -u
2020/06/16 16:28:32 logging.go:52 Command #1 took 876.419µs
2020/06/16 16:28:32 logging.go:37 Running command #2 /usr/bin/which which
2020/06/16 16:28:32 logging.go:52 Command #2 took 673.717µs
2020/06/16 16:28:32 logging.go:37 Running command #3 /usr/bin/which memsqlctl
2020/06/16 16:28:32 logging.go:52 Command #3 took 687.467µs
2020/06/16 16:28:32 logging.go:37 Running command #4 /usr/bin/memsqlctl --json --yes env
2020/06/16 16:28:32 logging.go:52 Command #4 took 5.341786ms
2020/06/16 16:28:32 logging.go:37 Running command #5 /usr/bin/which which
2020/06/16 16:28:32 logging.go:52 Command #5 took 683.233µs
2020/06/16 16:28:32 logging.go:37 Running command #6 /usr/bin/which sudo
2020/06/16 16:28:32 logging.go:52 Command #6 took 650.381µs
2020/06/16 16:28:32 logging.go:37 Running command #7 /usr/bin/sudo -n -k id
2020/06/16 16:28:32 logging.go:52 Command #7 failed with exit code 1 in 3.703279ms
2020/06/16 16:28:32 logging.go:37 Running command #8 /usr/bin/id -u -n
2020/06/16 16:28:32 logging.go:52 Command #8 took 1.075713ms
The target user user on host 127.0.0.1 does not have the privileges to perform this action.
Please enter your password to proceed with sudo. (For details, see SingleStore Tools Issues · SingleStore Documentation).
sudo password for user@127.0.0.1:
2020/06/16 16:28:37 logging.go:37 Running command #9 /usr/bin/timeout --foreground 5 sudo -S -k -v
2020/06/16 16:28:37 logging.go:52 Command #9 took 10.801885ms
Toolbox is about to perform the following actions on host 127.0.0.1:
· Run ‘memsqlctl create-node --port 3307 --base-install-dir /misc/memsql/master --password ●●●●●●’

Would you like to continue? [y/N]: y
2020/06/16 16:28:40 logging.go:37 Running command #10 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 582f967c-dae7-442f-be88-896fa8156545 && exec memsqlctl --json --yes upsert-secure-key
2020/06/16 16:28:40 logging.go:52 Command #10 took 16.579303ms
2020/06/16 16:28:40 logging.go:37 Running command #11 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 582f967c-dae7-442f-be88-896fa8156545 && exec memsqlctl --json --yes create-node --base-install-dir /misc/memsql/master --port 3307 --secure-password SmiiczgSopoAr6ft1g7IWeov5r7hgXhP673i/w==
2020/06/16 16:28:41 logging.go:52 Command #11 failed with exit code 1 in 722.629576ms
error running memsqlctl: error running command: /usr/bin/sudo -S -k -- /bin/sh -c ''\''printf'\'' '\''582f967c-dae7-442f-be88-896fa8156545'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''create-node'\'' '\''--base-install-dir'\'' '\''/misc/memsql/master'\'' '\''--port'\'' '\''3307'\'' '\''--secure-password'\'' '\''SmiiczgSopoAr6ft1g7IWeov5r7hgXhP673i/w=='\''': exit status 1
stderr: [sudo] password for user: Latest errors from MemSQL tracelog:
49 2020-06-16 16:28:41.232 INFO: Log opened
: Failed to connect to MemSQL: Check to see if another instance of MemSQL is already running on port 3307

memsql-admin create-node --host 127.0.0.1 --password "" -P 3317 --base-install-dir /misc/memsql2/master -vvv

2020/06/16 16:29:46 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/16 16:29:46 env.go:319 Toolbox version: 1.5.5
2020/06/16 16:29:46 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox853631201 SSHBinaryPath:/usr/bin/ssh}
2020/06/16 16:29:46 logging.go:37 Running command #1 /usr/bin/id -u
2020/06/16 16:29:46 logging.go:52 Command #1 took 834.252µs
2020/06/16 16:29:46 logging.go:37 Running command #2 /usr/bin/which which
2020/06/16 16:29:46 logging.go:52 Command #2 took 690.437µs
2020/06/16 16:29:46 logging.go:37 Running command #3 /usr/bin/which memsqlctl
2020/06/16 16:29:46 logging.go:52 Command #3 took 686.747µs
2020/06/16 16:29:46 logging.go:37 Running command #4 /usr/bin/memsqlctl --json --yes env
2020/06/16 16:29:46 logging.go:52 Command #4 took 5.170339ms
2020/06/16 16:29:46 logging.go:37 Running command #5 /usr/bin/which which
2020/06/16 16:29:46 logging.go:52 Command #5 took 688.313µs
2020/06/16 16:29:46 logging.go:37 Running command #6 /usr/bin/which sudo
2020/06/16 16:29:46 logging.go:52 Command #6 took 713.135µs
2020/06/16 16:29:46 logging.go:37 Running command #7 /usr/bin/sudo -n -k id
2020/06/16 16:29:46 logging.go:52 Command #7 failed with exit code 1 in 3.670691ms
2020/06/16 16:29:46 logging.go:37 Running command #8 /usr/bin/id -u -n
2020/06/16 16:29:46 logging.go:52 Command #8 took 1.043561ms
The target user user on host 127.0.0.1 does not have the privileges to perform this action.
Please enter your password to proceed with sudo. (For details, see SingleStore Tools Issues · SingleStore Documentation).
sudo password for user@127.0.0.1:
2020/06/16 16:29:50 logging.go:37 Running command #9 /usr/bin/timeout --foreground 5 sudo -S -k -v
2020/06/16 16:29:50 logging.go:52 Command #9 took 10.621054ms
Toolbox is about to perform the following actions on host 127.0.0.1:
· Run ‘memsqlctl create-node --port 3317 --base-install-dir /misc/memsql2/master --password ●●●●●●’

Would you like to continue? [y/N]: y
2020/06/16 16:29:51 logging.go:37 Running command #10 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 613b9343-dda4-462a-b715-bc1567e81648 && exec memsqlctl --json --yes upsert-secure-key
2020/06/16 16:29:51 logging.go:52 Command #10 took 16.84353ms
2020/06/16 16:29:51 logging.go:37 Running command #11 /usr/bin/sudo -S -k -- /bin/sh -c $1
where $1 = printf 613b9343-dda4-462a-b715-bc1567e81648 && exec memsqlctl --json --yes create-node --base-install-dir /misc/memsql2/master --port 3317 --secure-password +Xouz+JKQaPBNejkLoTAE2UZVbFfOg5SfYHRFw==
2020/06/16 16:29:54 logging.go:52 Command #11 took 3.612578996s
±-----------------------------------------±----------+
| MemSQL ID | Host |
±-----------------------------------------±----------+
| 80DDF29585DA3F09A44FC27A229AF814D3663905 | 127.0.0.1 |
±-----------------------------------------±----------+

memsql-admin list-nodes

✘ Failed to list nodes on all hosts: failed to list nodes on host 127.0.0.1: error running memsqlctl: error running command: /usr/bin/memsqlctl --json --yes list-nodes: exit status 1
stderr: memsqlctl failed to load the MemSQL config file ‘/misc/memsql/master/memsql.cnf’ referenced by the node metadata file at ‘/var/lib/memsql/nodes.hcl’.
See update-config · SingleStore Documentation for more information.
No nodes found

less /var/lib/memsql/nodes.hcl

version = 1

node {
memsqlConfigPath = “/misc/memsql/master/memsql.cnf”

password {
cipherText = “iwR+3dNz5GY5qIwqFQ1hjBc60FX3jkwPtbGE/A==”
}
}

node {
memsqlConfigPath = “/misc/memsql2/master/memsql.cnf”

password {
cipherText = “nvNpW4B6o5ktExCjQzXauJCIR5MAzU9IgAyqdA==”
}
}

secure = “YF69WtEJHSJtM3FLKYcmfFV126Co2DY+EISVzYlCpAI=”

[ymonye]

I created a backup of the var/lib/memsql/nodes.hcl file & removed the lines for the failed node from port 3307.

node {
memsqlConfigPath = “/misc/memsql/master/memsql.cnf”

password {
cipherText = “iwR+3dNz5GY5qIwqFQ1hjBc60FX3jkwPtbGE/A==”
}
}

From there, I attempted to verify my nodes. I still received the same error (only for the node on port 3317):

memsql-admin list-nodes

✘ Failed to list nodes on all hosts: failed to list nodes on host 127.0.0.1: error running memsqlctl: error running command: /usr/bin/memsqlctl --json --yes list-nodes: exit status 1
stderr: memsqlctl failed to load the MemSQL config file ‘/misc/memsql2/master/memsql.cnf’ referenced by the node metadata file at ‘/var/lib/memsql/nodes.hcl’.
See update-config · SingleStore Documentation for more information.
No nodes found

[vtkachuk-ua]

Does file exist under the path: ‘/misc/memsql2/master/memsql.cnf’
If yes, what is the permission for it?

[ymonye]

# ls -la
total 28
drwxr-x---  6 memsql memsql 4096 Jun 16 16:29 .
drwxr-x---  3 memsql memsql 4096 Jun 16 16:29 ..
drwxr-x---  2 memsql memsql 4096 Jun 16 16:29 auditlogs
drwxr-x--- 11 memsql memsql 4096 Jun 16 16:29 data
-rw-r--r--  1 memsql memsql  441 Jun 16 16:29 memsql.cnf
drwxr-x--- 61 memsql memsql 4096 Jun 16 16:29 plancache
drwxr-x---  2 memsql memsql 4096 Jun 16 16:29 tracelogs

[yznovyak-ua]

Hi @ymonye! What package manager do you use? rpm? dpkg (deb files)? tar? But anyway, you can do the following:

1. Check out the contents of your /etc/memsql/memsqlctl.hcl. Particularly it has 2 entries: nodeMetadataFile and defaultInstallDir. Are those paths correct?

2. If those paths are correct, you should check the nodeMetadataFile (from the previous step). Particularly it should list a bunch of node { ... } structures and each of those should have memsqlConfigPath. Are those paths correct?

3. Are all of the paths owned by the memsql user and memsql group?

[ymonye]

Hey @yznovyak-ua, I used the package manager on a Deb system (Ubuntu 18.04.4).

less /etc/memsql/memsqlctl.hcl

version = 1
nodeMetadataFile = “/var/lib/memsql/nodes.hcl”
defaultInstallDir = “/var/lib/memsql”
user = “memsql”

less /var/lib/memsql/nodes.hcl

version = 1

node {
memsqlConfigPath = “/misc/memsql2/master/memsql.cnf”

password {
cipherText = “nvNpW4B6o5ktExCjQzXauJCIR5MAzU9IgAyqdA==”
}
}

secure = “YF69WtEJHSJtM3FLKYcmfFV126Co2DY+EISVzYlCpAI=”

/misc/memsql2/master/memsql.cnf

[server]
auditlogsdir = auditlogs
bind_address = 0.0.0.0
datadir = data
pid_file = memsqld.pid
plancachedir = plancache
port = 3317
tracelogsdir = tracelogs
basedir = .
core_file = true
lc_messages_dir = ./share
lock_wait_timeout = 60
snapshot_trigger_size = 2g
socket = memsql.sock
tmpdir = .

Everything is owned by the memsql user & group.

[yznovyak-ua]

Hm… Everything looks good there… Can you once again run memsqlctl list-nodes ?

If it fails, can you run memsqlctl list-nodes -vvv and post the output?

[ymonye]

memsqlctl list-nodes fails under my regular ‘user’ account, as well as memsql-admin list-nodes. However, I just noticed memsqlctl list-nodes works fine as my root user.

memsqlctl list-nodes -vvv

2020/06/17 20:35:16 state_manager.go:79 Acquiring shared lock on /var/lib/memsql/nodes.hcl.lock to permit non-exclusive read access to /var/lib/memsql/nodes.hcl
memsqlctl failed to load the MemSQL config file ‘/misc/memsql2/master/memsql.cnf’ referenced by the node metadata file at ‘/var/lib/memsql/nodes.hcl’.
See update-config · SingleStore Documentation for more information.

memsql-admin list-nodes -vvv

2020/06/17 20:37:10 backup_cache.go:43 Could not load backup cache from /home/user/.local/share/memsql-toolbox/backup-cache.hcl: could not read file /home/user/.local/share/memsql-toolbox/backup-cache.hcl: open /home/user/.local/share/memsql-toolbox/backup-cache.hcl: no such file or directory
2020/06/17 20:37:10 env.go:319 Toolbox version: 1.5.5
2020/06/17 20:37:10 env.go:320 Toolbox env: &{ConfigPath:/home/user/.config/memsql-toolbox/toolbox.hcl StateFilePath:/home/user/.local/share/memsql-toolbox/toolbox-state.hcl CacheFilePath:/home/user/.local/share/memsql-toolbox/toolbox-cache.hcl BackupCacheFilePath:/home/user/.local/share/memsql-toolbox/backup-cache.hcl RestoreBackupLockFilePath:/home/user/.cache/memsql-toolbox/restore.lock SessionDir:/run/user/1000/memsql-toolbox844363418 SSHBinaryPath:/usr/bin/ssh}
2020/06/17 20:37:10 logging.go:37 Running command #1 /usr/bin/id -u
2020/06/17 20:37:10 logging.go:52 Command #1 took 1.005028ms
2020/06/17 20:37:10 logging.go:37 Running command #2 /usr/bin/which which
2020/06/17 20:37:10 logging.go:52 Command #2 took 748.104µs
2020/06/17 20:37:10 logging.go:37 Running command #3 /usr/bin/which memsqlctl
2020/06/17 20:37:10 logging.go:52 Command #3 took 805.748µs
2020/06/17 20:37:10 logging.go:37 Running command #4 /usr/bin/memsqlctl --json --yes env
2020/06/17 20:37:10 logging.go:52 Command #4 took 5.153562ms
2020/06/17 20:37:10 logging.go:37 Running command #5 /usr/bin/id -G
2020/06/17 20:37:10 logging.go:52 Command #5 took 787.621µs
2020/06/17 20:37:10 logging.go:37 Running command #6 /usr/bin/memsqlctl --json --yes list-nodes
2020/06/17 20:37:10 logging.go:52 Command #6 failed with exit code 1 in 4.897486ms
✘ Failed to list nodes on all hosts: failed to list nodes on host 127.0.0.1: error running memsqlctl: error running command: /usr/bin/memsqlctl --json --yes list-nodes: exit status 1
stderr: memsqlctl failed to load the MemSQL config file ‘/misc/memsql2/master/memsql.cnf’ referenced by the node metadata file at ‘/var/lib/memsql/nodes.hcl’.
See update-config · SingleStore Documentation for more information.
No nodes found

LOGGED IN AS ROOT

memsql-admin list-nodes

Toolbox does not support implicitly running as root. Please add ‘user = “root”’ to /root/.config/memsql-toolbox/toolbox.hcl if you want to run as the root user. See Toolbox · SingleStore Documentation
root@yonniq:/var/lib/memsql# memsqlctl list-nodes
±-----------±--------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| MemSQL ID | Role | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
±-----------±--------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| 80DDF29585 | Unknown | 3317 | Running | True | 7.1.3 | Online | | 0.0.0.0 |
±-----------±--------±-----±--------------±-------------±--------±---------------±-------------------±-------------+

[ymonye]

Also, the /var/lib/memsql/nodes.hcl.lock file is empty.

[yznovyak-ua]

Sorry for the delay.

The fact that memsqlctl list-nodes works with sudo but doesn’t work with you regular user suggest that something is off with the permissions.

What user are you running your memsqlctl list-nodes queries as?

Can you say who owns (what user and what group) and what the permissions (you can get that info from ls -la <parent_dir>) for the following directories:

• /misc
• /misc/memsql2
• /misc/memsql2/master
• /misc/memsql2/master/memsql.cnf

In the following format, if possible:

drwxr-xr-x   2 yznovyak-ua yznovyak-ua   4096 Jun 17 22:09 memsql

Also, does /misc happen to be mounted somewhere else? If yes, what is the FS?

(I asked about memsqlctl list-nodes even though you were running memsql-admin list-nodes; that’s intentional because according to -vvv your memsql-admin runs memsqlctl list-nodes (this is by design))

[ymonye]

memsqlctl list-nodes was run as both my ‘user’ & ‘root’ usernames. It only successfully ran from the ‘root’ username. /misc is a mounted RAID array, with the ext4 filesystem. This is what I had & verified successfully working before the old SSD crashed.

ls -la /misc

total 60
drwx------ 8 memsql memsql 4096 Jun 18 20:59 .
drwxr-xr-x 33 root root 4096 Jun 16 16:27 …
drwxr-xr-x 2 memsql memsql 12288 Jun 7 14:01 db_20200607_leaf
drwxr-xr-x 2 memsql memsql 4096 Jun 7 14:05 db_20200607_master
drwxr-xr-x 3 memsql memsql 12288 Jun 12 06:50 db_20200612
drwx------ 2 root root 16384 Jun 1 12:48 lost+found
drwxr-x— 3 memsql memsql 4096 Jun 16 16:28 memsql
drwxr-x— 3 memsql memsql 4096 Jun 16 16:29 memsql2

ls -la /misc/memsql2

total 12
drwxr-x— 3 memsql memsql 4096 Jun 16 16:29 .
drwx------ 8 memsql memsql 4096 Jun 18 20:59 …
drwxr-x— 6 memsql memsql 4096 Jun 16 16:29 master

ls -la /misc/memsql2/master/

total 28
drwxr-x— 6 memsql memsql 4096 Jun 16 16:29 .
drwxr-x— 3 memsql memsql 4096 Jun 16 16:29 …
drwxr-x— 2 memsql memsql 4096 Jun 16 16:29 auditlogs
drwxr-x— 11 memsql memsql 4096 Jun 18 20:30 data
-rw-r–r-- 1 memsql memsql 441 Jun 16 16:29 memsql.cnf
drwxr-x— 63 memsql memsql 4096 Jun 17 20:33 plancache
drwxr-x— 2 memsql memsql 4096 Jun 16 16:29 tracelogs

ls -la /misc/memsql2/master/memsql.cnf

-rw-r–r-- 1 memsql memsql 441 Jun 16 16:29 /misc/memsql2/master/memsql.cnf

[yznovyak-ua]

Oh! It looks like /misc/memsql2/master has a 750 permissions (user: rwx, group: r-x, other: ---). Hence if you run memsqlctl as a user you can’t enumerate files in that directory and hence can’t read memsql.cnf from it.

You should add your user into the memsql group:

usermod -a -G memsql user

(you may need to run it with sudo)

[yznovyak-ua]

Thanks for your question – we will update that error message to make it easier to figure out what was the issue.

[ymonye]

@yznovyak-ua, but memsql.cnf with a permission setting of 750 is normal, as that’s what we have on a couple of other machines of ours. On those machines, the ‘user’ account is also able to access memsql-admin without any issues.

So back to this machine, even after adding user to the memsql group (which most likely was already done from the get-go), I still receive that error.

memsql-admin list-nodes

✘ Failed to list nodes on all hosts: failed to list nodes on host 127.0.0.1: error running memsqlctl: error running command: /usr/bin/memsqlctl --json --yes list-nodes: exit status 1
stderr: memsqlctl failed to load the MemSQL config file ‘/misc/memsql2/master/memsql.cnf’ referenced by the node metadata file at ‘/var/lib/memsql/nodes.hcl’.
See update-config · SingleStore Documentation for more information.
No nodes found

I even went ahead and changed all file permissions in the memsql2/ directory to 777 and still had the above error.

ls -la

total 28
drwxrwxrwx 6 memsql memsql 4096 Jun 16 16:29 .
drwxrwxrwx 3 memsql memsql 4096 Jun 16 16:29 …
drwxrwxrwx 2 memsql memsql 4096 Jun 16 16:29 auditlogs
drwxrwxrwx 11 memsql memsql 4096 Jun 18 23:23 data
-rwxrwxrwx 1 memsql memsql 441 Jun 16 16:29 memsql.cnf
drwxrwxrwx 63 memsql memsql 4096 Jun 17 20:33 plancache
drwxrwxrwx 2 memsql memsql 4096 Jun 16 16:29 tracelogs

[ymonye]

Ahhhh, looks like I found the issue (d’oh!)

The main /misc directory had permissions set to 700. After changing that to 755, I was able to run memsql-admin without any issue. Thanks so much for your & @vtkachuk-ua’s help!

drwx------ 7 memsql memsql 4096 Jun 18 23:37 misc

[yznovyak-ua]

Oh, sorry I missed the /misc thing. But anyway, glad that got resolved. We’ll update that error message to say why did memsqlctl fail to open the file (in this case it will likely say permission denied).

[ymonye]

Sounds good, thanks!