View unanswered posts    View active topics

All times are UTC - 6 hours





Post new topic Reply to topic  [ 25 posts ] 
Go to page 1, 2  Next

Print view Previous topic   Next topic  
Author Message
Search for:
PostPosted: Wed Nov 09, 2016 8:42 am 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
So, I seem never to have had a smooth upgrade process. I've got a dedicated LinHES system, partitioned just the way it wants it to be. My manipulations of the system have been along the lines of editing a few scripts, updating configuration files, and patching some .xml files in the mythtv themes. Nothing very intrusive.

Because I was upgrading from 8.2 to 8.4.1, I created an ISO on a DVD (my BIOS cannot boot from USB). I booted it up, and ran the upgrade procedure. It reformatted my root partition, completed the package installs, then the screen showed that sysconfig failed, and the system went quiet. I went into a text VT and looked for log information to tell me what went wrong with sysconfig, but I couldn't find it anywhere. The system I was left with could boot, but it had lost its entire personality.

It took me an hour to recover from backup. The DVD doesn't seem to detect my USB keyboard, so in the first menu, when it offers me options, my only choice is to wait until it times out and boots the default selection. Since I was using the DVD as a recovery system, that made the restore painfully slow, every time I rebooted to see if I had managed to repair the system. I reformatted the root partition and copied back the files from backup.

Because the root partition was reformatted and the UUID changed, my backed up system couldn't find that disc anymore, until I used tune2fs to reset the UUID to the value it had before the upgrade began. The attempted upgrade had also overwritten the grub bootloader with a version that was inconsistent with the old backup, so I had to chroot into the boot disc and re-run grub-install (first, you have to mount various pseudo-mounts like proc, dev/pty, sys, and so on or you get cryptic failures).

At this point, I could boot the system, but the backend wouldn't come up. It took me a long time to figure out that, somehow, the mythtv (unprivileged) user had lost the capability to open ICMP sockets. I used strace and reviewed a dozen files before I saw that there was a spot that did a fork/exec on ping, and that was error-ing out. I remembered that the configuration file had an entry to ping the DB host, so I set that to zero, and the system was back to usable.

So, what's my next step? How can I figure out why the sysconfig failed? What's up with the capabilities, how did they get changed?


Top
 Profile  
 
PostPosted: Wed Nov 09, 2016 12:22 pm 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
Format the root partition with another linux ISO. The LinHES iso doesn't do a good job of formatting the drive in my experience. Try installing fresh.

_________________
My System


Top
 Profile  
 
PostPosted: Wed Nov 09, 2016 12:48 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
mattbatt wrote:
Format the root partition with another linux ISO. The LinHES iso doesn't do a good job of formatting the drive in my experience. Try installing fresh.

The upgrade procedure reports that it does mke2fs -j on the root partition. I'm not sure how we can get much more reformatted than that.

When examining the carnage after the upgrade, I found that the root partition contained an /etc and an /etc.old. Presumably, the /etc.old was recovered from a backup taken by the upgrade process. If necessary, I can manually copy over everything I think needs to be moved from /etc.old to /etc, but this is an automated process used by many before me, it should have worked. I noticed, though, that there was no /root.old, and the /root directory was not repopulated. That's where I keep many of my site-specific scripts (going back to the time when mythbackend ran as root, so it made sense to put its scripts in /root/bin).


Top
 Profile  
 
PostPosted: Sun Nov 13, 2016 12:10 am 
Offline
Joined: Tue Aug 15, 2006 11:14 am
Posts: 1343
Location: Orlando FL
All I know is on two separate iso upgrades I had to nuke the partition with Ubuntu live disk and that magically fixed my lingering problems. Maybe this upgrade they fixed it in not sure but I still don't trust it.

_________________
My System


Top
 Profile  
 
PostPosted: Mon Nov 14, 2016 2:03 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
mattbatt wrote:
All I know is on two separate iso upgrades I had to nuke the partition with Ubuntu live disk and that magically fixed my lingering problems. Maybe this upgrade they fixed it in not sure but I still don't trust it.


I'm not clear on how you reformatted the partition but still managed an "upgrade". The reformatting ought to have deleted the entire personality.

So, I've been trying to make progress on this. Rather than scrambling my MythTV box, I created a virtual machine copy of it under qemu. Exactly the same disc layout, mount points, UUIDs, disc labels, etc. Then I tried to run the upgrade ISO on it. Strangely, it hangs part-way through the mke2fs. The result is a completely corrupted root partition on the virtual machine. Also annoying, the virtual terminals aren't available in the virtual machine when the graphics have started up. The virtual terminals are there when running on the real system. I'll have to use the ALT-X xterms to look around.

One thing I did notice is that there was no backup of the important data on my /myth partition before the sda1 disc was formatted. That could mean that the backup data was in the virtual DVD filesystem, so in RAM. That's a touch dangerous, but as I have complete backups, not critical.


Top
 Profile  
 
PostPosted: Sat Nov 19, 2016 10:19 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
OK, I had a chance to make another attempt today. I wasn't planning to see it succeed, but I could at least examine the debris. I took images of the DVD COW system and the failed upgraded /dev/sda1 before reformatting and restoring /dev/sda1. That's the nice thing about having a MythTV box, you probably have hundreds of GB of free space sitting around that can be used for such examinations.

The upgrade script failed because systemconfig failed to find "default DHCP". Looking in the /tmp/mv_debug.log in the live DVD system, I see the following (reformatted for clarity):
Code:
11-20 03:24 - root - DEBUG -     (256, '
----------------------start of systemconfig all,this_is_install ----------------------
cat: /usr/local/share/mythtv/.releasetype: No such file or directory
    *Error occured finding default dhcp
Traceback (most recent call last):
  File "/usr/MythVantage/bin/systemconfig.py", line 462, in <module>
    main(sys.argv[1:])
  File "/usr/MythVantage/bin/systemconfig.py", line 239, in main
    systemconfig["mythip"] = mv_network.setup_MYTH_IP(systemconfig)
  File "/usr/MythVantage/bin/mv_network.py", line 95, in setup_MYTH_IP
    defaultip = get_ip(default_interface)
  File "/usr/MythVantage/bin/mv_network.py", line 65, in get_ip
    struct.pack(\'256s\', ifname[:15])
IOError: [Errno 99] Cannot assign requested address
systemconfig.py exit code 1')

According to the logs, the error occurred while it was running in a chroot, so I checked the /tmp directory on the image of the failed upgraded sda1. This was in systemconfig_debug.log:
Code:
11-20 03:24 - root - DEBUG - ______START OF DEBUG______
11-20 03:24 - root - CRITICAL -     *Error occured finding default dhcp
11-20 03:24 - root - DEBUG -     Using 0 as dhcp value for eth0
11-20 03:24 - root - DEBUG -    dhcp is in use, finding dhcp ip
11-20 03:24 - root - DEBUG -     Finding ip address for eth0
11-20 03:24 - root - DEBUG -     found interfaces:['eth0']
11-20 03:24 - root - DEBUG -     Found eth0 in all_interfaces

Looking at the mv_network.py in the upgraded sda1, line 65 is where the code is trying to use an ioctl to retrieve the IP number of eth0. It is not clear why this results in a "Cannot assign requested address" error, or what I can do to alleviate this.

Perhaps I should hard-code the IP number in my pre-upgraded box, setting it to that value that it would have obtained from DHCP, and see if the upgrade will work on a machine with a non-DHCP-assigned address.

I welcome advice or opinions.


Top
 Profile  
 
PostPosted: Mon Nov 21, 2016 7:51 am 
Offline
Joined: Fri Jul 21, 2006 11:12 pm
Posts: 1194
Location: SC
I just found a similar issue when doing an upgrade on a box that isn't connected to the network. Although it fails a few lines earlier. I am guessing you are connected to the network as it did find eth0?

I am not sure this is exactly correct but it is worth testing. Boot the live ISO and before running the upgrade open a term and edit /usr/MythVantage/bin/mv_network.py and replace line 62-66 with:
Code:
    try:
        ip = socket.inet_ntoa(fcntl.ioctl(
            s.fileno(),
            0x8915,  # SIOCGIFADDR
            struct.pack('256s', ifname[:15])
        )[20:24])
    except:
        ip = "127.0.0.1"

Then run the upgrade.


Top
 Profile  
 
PostPosted: Mon Nov 21, 2016 9:40 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
brfransen wrote:
I just found a similar issue when doing an upgrade on a box that isn't connected to the network. Although it fails a few lines earlier. I am guessing you are connected to the network as it did find eth0?

Yes, that's right. I'm on a network, with address obtained through DHCP.

brfransen wrote:
I am not sure this is exactly correct but it is worth testing. Boot the live ISO and before running the upgrade open a term and edit /usr/MythVantage/bin/mv_network.py and replace line 62-66 with:
Code:
    try:
        ip = socket.inet_ntoa(fcntl.ioctl(
            s.fileno(),
            0x8915,  # SIOCGIFADDR
            struct.pack('256s', ifname[:15])
        )[20:24])
    except:
        ip = "127.0.0.1"

Then run the upgrade.


I thought of that, but that won't work. My failure happens in the chroot. That means it's in the mv_network.py file that is unpacked onto /dev/sda1 during the upgrade, not the version that resides on the live DVD. I would have to edit the contents of the .iso and burn a new DVD. I believe, though, that the entire function is skipped if we have a statically-assigned IP number, rather than one picked up from DHCP. I'll try that the next time I have a hole in the schedule.


Top
 Profile  
 
PostPosted: Tue Nov 22, 2016 7:31 am 
Offline
Joined: Fri Jul 21, 2006 11:12 pm
Posts: 1194
Location: SC
neufeld wrote:
I thought of that, but that won't work. My failure happens in the chroot. That means it's in the mv_network.py file that is unpacked onto /dev/sda1 during the upgrade, not the version that resides on the live DVD. I would have to edit the contents of the .iso and burn a new DVD.
Not true. It will work and does work because /usr/MythVantage/bin is copied to /new_boot (the /dev/sda1 mount) before systemconfig is run in the chroot.


Top
 Profile  
 
PostPosted: Fri Nov 25, 2016 10:46 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
OK, latest news. I applied the patch suggested. It did allow the upgrade to complete. However, the resulting system was unusable. It wouldn't boot up to graphics, and I'm not familiar with how systemd logs its problems. I tried to fix what I saw on VT1 during multiple serial boots. Here are some of the things that were wrong:

    Hostname was incorrect.
    The eth0 interface was not asked to run dhcpcd, so there was no networking.
    The mount point /myth did not exist, so it couldn't mount the media drives.
    The formerly existing mount point /data/storage/OCZ-VERTEX3_OCZ-2Z3EE2ZBB30TF4QY did not exist, causing a bind mount failure
    The /etc/localtime symlink had been changed from EST5EDT to Samoa
    There were logging errors related to missing groups: systemd-network and systemd-journal-remote, neither of which existed
    There were logging errors related to a missing user: systemd-network
    The irexec service was looping forever because there was no /etc/lircrc file.
    The /etc/machine-id file was empty, causing logging errors due to a failure to expand a %m directive

I poked at it for about an hour, but it was still hanging on VT1 with no on-screen indication of why it wasn't continuing.

At this point, it looks like I'd be better off doing a clean install and then resetting the machine's personality, database, and media directories. This includes configuring the lirc setup I have to run an IR blaster (I don't use lirc for controlling the myth box, I use it only to command the STBs). That's a major project though, and it's hard to set aside enough time when the machine isn't busy recording and when the baby is asleep.


Top
 Profile  
 
PostPosted: Sat Nov 26, 2016 7:49 am 
Offline
Joined: Fri Jul 21, 2006 11:12 pm
Posts: 1194
Location: SC
Many of those things (hostname, mounts, timezone, lircrc, machine-id) suggest that systemconfig.py didn't work on the chroot. Anything in the /tmp/systemconfig_debug.log?

A couple of things:
- It could be that the correct graphics driver didn't get installed. Try running xconfig.sh from the VT. Does it seem to be detecting your card correctly?
- We don't use systemd for booting or much at all. It is there because some packages have started requiring it. But the logging errors are odd. Any idea what was trying to run that triggered that error?
- /myth is not a mount but a symlink to /dev/storage/disk0/media. It being missing shouldn't cause the drives not to mount unless you had manually setup your mounts that way. add_storage.py should have recreated your mounts at /data/storage from the config files in /etc/storage.d/.


Top
 Profile  
 
PostPosted: Sat Nov 26, 2016 4:17 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
brfransen wrote:
Many of those things (hostname, mounts, timezone, lircrc, machine-id) suggest that systemconfig.py didn't work on the chroot. Anything in the /tmp/systemconfig_debug.log?

I'm attaching the logs from the chroot /tmp. There's a lot of content there, it seems to think it did things.
I set up the mv_network.py to assign 192.168.1.8, which is the address the MythTV box gets from DHCP.

brfransen wrote:
A couple of things:
- It could be that the correct graphics driver didn't get installed. Try running xconfig.sh from the VT. Does it seem to be detecting your card correctly?

I doubt this is a problem, it's a few-year-old NVidia card. VT1 didn't get a getty, so it's not that X failed to start, but that it was still running the startup scripts, and was hanging. Fortunately there was a getty on VT2, so I could look around and copy the image away before restoring from backup. The xorg.conf setup in the failed upgrade is configured for NVidia.
brfransen wrote:
- We don't use systemd for booting or much at all. It is there because some packages have started requiring it. But the logging errors are odd. Any idea what was trying to run that triggered that error?

No, I was just trying to repair damage quickly and get a go/no-go on this before bed time. I didn't write down much.
brfransen wrote:
- /myth is not a mount but a symlink to /dev/storage/disk0/media. It being missing shouldn't cause the drives not to mount unless you had manually setup your mounts that way. add_storage.py should have recreated your mounts at /data/storage from the config files in /etc/storage.d/.

Yes, I manually set my mounts, many years ago, and they just continued through upgrades. I have three 3TB drives. One mounted on /myth, the other two mounted underneath that, on one /myth/tv2 and the other on /myth/tv3.

When I next have time to work on this, I'm going to be doing a fresh install in a VM and personalizing that. Once it looks happy, I'll copy everything over to the Myth box and see if that works.


Attachments
File comment: Logs in the chroot following the nominally successful upgrade from 8.2 to 8.4.1.
chroot-tmp-logs.tgz [13.49 KiB]
Downloaded 587 times
Top
 Profile  
 
PostPosted: Sat Nov 26, 2016 5:38 pm 
Offline
Joined: Fri Jul 21, 2006 11:12 pm
Posts: 1194
Location: SC
Lots of things failing in there but the main things I see is that mysql won't start and the runit services are trying to start but fail. Does the /etc/sv dir exist?


Top
 Profile  
 
PostPosted: Sat Nov 26, 2016 10:01 pm 
Offline
Joined: Sun Apr 30, 2006 7:49 pm
Posts: 37
Location: Ottawa
brfransen wrote:
Lots of things failing in there but the main things I see is that mysql won't start and the runit services are trying to start but fail. Does the /etc/sv dir exist?

There is no /run/mysqld in the chroot, so even if mysqld is running, nobody can contact it through UNIX sockets. The /etc/sv dir is present, and populated with 75 directories, so that part seems OK. Only a handful of them have subdirectories called "supervise", though.


Top
 Profile  
 
PostPosted: Sun Nov 27, 2016 7:27 am 
Offline
Joined: Fri Jul 21, 2006 11:12 pm
Posts: 1194
Location: SC
I wonder if the copy is failing before systemconfig runs. Maybe a bad ISO burn. Did you confirm the md5sum of the burn?
In the chroot /data/srv/mysql/ is there any *.err files that give any hint of why mysql isn't starting?


Top
 Profile  
 

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 25 posts ] 
Go to page 1, 2  Next



All times are UTC - 6 hours




Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group

Theme Created By ceyhansuyu