Every Sunday morning for the last 3 weeks at 4:22 I recieve 2 emails stating that my raid is rebuilding and then the rebuild is complete. Email #1: This is an automatically generated mail message from mdadm running on mydomain.com. A RebuildStarted event has been detected on md device /dev/md1. Email #2: This is an automatically generated mail message from mdadm running on mydomain.com. A RebuildFinished event has been detected on md device /dev/md1. The event is complete within seconds: Aug 8 04:22:01 mail kernel: md: syncing RAID array md1 Aug 8 04:22:01 mail kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. Aug 8 04:22:01 mail kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction. Aug 8 04:22:01 mail kernel: md: using 128k window, over a total of 104320 blocks. Aug 8 04:22:01 mail kernel: klogd 1.4.1, ---------- state change ---------- Aug 8 04:22:01 mail kernel: Inspecting /boot/System.map-2.6.18-194.8.1.el5PAE Aug 8 04:22:01 mail kernel: Loaded 30046 symbols from /boot/System.map-2.6.18-194.8.1.el5PAE. Aug 8 04:22:01 mail kernel: Symbols match kernel version 2.6.18. Aug 8 04:22:01 mail kernel: No module symbols loaded - kernel modules not enabled. Aug 8 04:22:05 mail kernel: md: md1: sync done. Aug 8 04:22:05 mail kernel: RAID1 conf printout: Aug 8 04:22:05 mail kernel: --- wd:3 rd:3 Aug 8 04:22:05 mail kernel: disk 0, wo:0, o:1, dev:sda1 Aug 8 04:22:05 mail kernel: disk 1, wo:0, o:1, dev:sdb1 Aug 8 04:22:05 mail kernel: disk 2, wo:0, o:1, dev:sdc1 Also originally set this up as 2 disk mirror plus spare. Is this normal to only have the spare on md2? If not normal, is it related or a separate issue? [root@mail ~]# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdc1[2] sdb1[1] sda1[0] 104320 blocks [3/3] [UUU] md2 : active raid1 sdc2[2](S) sdb2[1] sda2[0] 35800768 blocks [2/2] [UU] unused devices: <none>
Released as an update 23/24 July 13 Closing.
*** Bug 7135 has been marked as a duplicate of this bug. ***
= RESOLUTION: Fixed in e-smith-base-5.2.0-78.el5.sme * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme - Prevent emailing about the normal, weekly, checks of RAID array, by Mark Casey [SME: 6160] = VERIFIED OR REOPEN: Jun 30 04:22:01 test81x kernel: md: syncing RAID array md1 Jun 30 04:22:05 test81x kernel: md: syncing RAID array md2 No warning email. VERIFIED = DOCUMENTATION IMPACT: Document the fact that a Periodic scrub of RAID arrays is performed every week on Sunday at 04:22 local time, refer Bug #3535. Theses operations are logged, however, no email are sent to admin. = SUGGESTED RELEASE NOTES: Prevent emailing about the normal, weekly, checks of RAID array
(In reply to Ian Wells from comment #112) > Would the following be a more accurate changelog? > > Fixed in e-smith-base-5.2.0-78.el5.sme > > * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme > - Prevent emailing about the normal, weekly, checks of RAID arrays, > by Mark Casey [SME: 6160] It is. 1) I would prefer waiting for the cron job on Sunday before resolving verified, however, it looks like the warning issue is fixed. I will follow up over the weekend. 2) The next step is to schedule the "check" at a different time so as not to > > conflict with clamscan, refer Bug #7656 3) A NFR should be raised to template /etc/sysconfig/raid_check and although the default would be ENABLED=yes, it could be set to ENABLED=no.
Would the following be a more accurate changelog? Fixed in e-smith-base-5.2.0-78.el5.sme * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme - Prevent emailing about the normal, weekly, checks of RAID arrays, by Mark Casey [SME: 6160]
(In reply to chris burnat from comment #105) > (In reply to Filippo Carletti from comment #104) > > While the fix is right, the changelog is misleading. > > The patch does NOT prevent weekly rebuilds of arrays, it disables > > notification mails about rebuilds (checks by default). > > Arrays are still checked for errors every sunday. > > Thanks Filippo. If the case, this is akin to putting our head in the sand, > and not what is required. Does your patch actually prevent the weekly > rebuilds? Excuse my ignorance. So is this bug effectively about preventing emails to admin about the RAID when all is well - so stop freaking out the admin. However in the background the checks continue and any real problems are emailed to admin. The comments suggest that only idle I?O is used for the check and that they are beneficial, however other comments suggest that it is undesirable to run this weekly and that there should be an option to disable it. If I have understood then this bug should only consider the sending of the email (and the changelog can be updated in CVS). A NFR could be raised to template /etc/sysconfig/raid_check and although the default would be ENABLED=yes, it could be set to ENABLED=no.
Thanks for the tip, effectively the raid event starts a little bit after 4.00AM. This is what i can found in /var/log/messages, but effectively i haven't got any more mails about this reconstruction. As i can understand philipo is suggesting that the weekly array checks is really important. Therefore there are two ways of thinking, because you feel that the load and the HD works by the check array are not normal and not wanted. Of course my array is little since It is a Test VM ------------------------------------------------------------------------------ Jun 30 04:22:02 sme8dev64 kernel: md: syncing RAID array md1 Jun 30 04:22:02 sme8dev64 kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. Jun 30 04:22:02 sme8dev64 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction. Jun 30 04:22:02 sme8dev64 kernel: md: using 128k window, over a total of 104320 blocks. Jun 30 04:22:02 sme8dev64 kernel: md: delaying resync of md2 until md1 has finished resync (they share one or more physical units) Jun 30 04:22:03 sme8dev64 kernel: md: md1: sync done. Jun 30 04:22:03 sme8dev64 kernel: md: syncing RAID array md2 Jun 30 04:22:03 sme8dev64 kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. Jun 30 04:22:03 sme8dev64 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction. Jun 30 04:22:03 sme8dev64 kernel: md: using 128k window, over a total of 52323584 blocks. Jun 30 04:22:03 sme8dev64 kernel: RAID1 conf printout: Jun 30 04:22:03 sme8dev64 kernel: --- wd:2 rd:2 Jun 30 04:22:03 sme8dev64 kernel: disk 0, wo:0, o:1, dev:sda1 Jun 30 04:22:03 sme8dev64 kernel: disk 1, wo:0, o:1, dev:sdb1 Jun 30 04:28:14 sme8dev64 kernel: md: md2: sync done. Jun 30 04:28:14 sme8dev64 kernel: RAID1 conf printout: Jun 30 04:28:14 sme8dev64 kernel: --- wd:2 rd:2 Jun 30 04:28:14 sme8dev64 kernel: disk 0, wo:0, o:1, dev:sda2 Jun 30 04:28:14 sme8dev64 kernel: disk 1, wo:0, o:1, dev:sdb2 (END)
You will be able to get the performance result from your system log. You can also run 'watch cat /proc/mdstat' which will give good 'real time' This patch will seem suitable because it is a small raid1, a real test is a large raid5, and while it is running do some moderate database type activity. MyOB would be severely impacted.
I have tried the patch on a full upgrade sme8 both 64 and 32 bit version [root@sme8dev ~]# rpm -qa e-smith-base e-smith-base-5.2.0-79.el5.sme [root@sme8dev64 ~]# rpm -qa e-smith-base e-smith-base-5.2.0-79.el5.sme i have modified the time of my server to be few minutes before sunday 4.00AM and i can hear the hd working, but effectively i haven't get messages by mail as before. do you have a way to see if mdadm is working and what it is doing, because I have tried this without success [root@sme8dev64 ~]# mdadm --detail /dev/md1 /dev/md1: Version : 0.90 Creation Time : Sat Feb 2 23:20:19 2013 Raid Level : raid1 Array Size : 104320 (101.89 MiB 106.82 MB) Used Dev Size : 104320 (101.89 MiB 106.82 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Sun Jun 30 04:02:45 2013 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : a62a1e05:f01bf65a:c4c71996:c4d5fe58 Events : 0.4 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 ---------------------------------------------------------------------- [root@sme8dev64 ~]# mdadm --detail /dev/md2 /dev/md2: Version : 0.90 Creation Time : Sat Feb 2 23:20:20 2013 Raid Level : raid1 Array Size : 52323584 (49.90 GiB 53.58 GB) Used Dev Size : 52323584 (49.90 GiB 53.58 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Sun Jun 30 04:07:10 2013 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : 6a5014da:068049bb:16c72cdd:bdf5a674 Events : 0.28 Number Major Minor RaidDevice State 0 8 2 0 active sync /dev/sda2 1 8 18 1 active sync /dev/sdb2 --------------------------------------------------------------------- [root@sme8dev64 ~]# cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU] md2 : active raid1 sdb2[1] sda2[0] 52323584 blocks [2/2] [UU] unused devices: <none>
Chris, I think your comment is correct - this is head in the sand stuff The load of the check placed on the server is too huge to be of benefit, it's an upstream bug, this only suppresses the messages. It's of little consequence if the disks involved a less than .. say 500g, but above 1tb it's several hours of hard disk grind. The correct way would be to template /etc/sysconfig/raid_check to allow ENABLED to be set to 'no'
Weekly array checks are vital. :-) See bug 3535 for some background. Recovery is painful.
(In reply to Filippo Carletti from comment #104) > (In reply to Ian Wells from comment #99) > > Fixed in e-smith-base-5.2.0-78.el5.sme > > > > * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme > > - Prevent weekly rebuilds of RAID arrays, by Mark Casey [SME: 6160] > > While the fix is right, the changelog is misleading. > The patch does NOT prevent weekly rebuilds of arrays, it disables > notification mails about rebuilds (checks by default). > Arrays are still checked for errors every sunday. Thanks Filippo. If the case, this is akin to putting our head in the sand, and not what is required. Does your patch actually prevent the weekly rebuilds? Excuse my ignorance.
(In reply to Ian Wells from comment #99) > Fixed in e-smith-base-5.2.0-78.el5.sme > > * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme > - Prevent weekly rebuilds of RAID arrays, by Mark Casey [SME: 6160] While the fix is right, the changelog is misleading. The patch does NOT prevent weekly rebuilds of arrays, it disables notification mails about rebuilds (checks by default). Arrays are still checked for errors every sunday.
Bug 6160 - Raid rebuild runs every week VERIFICATION- part 1 = ENVIRONMENT: sme8 clean + all updates. = ORIGINAL PROBLEM: Raid rebuild runs every week at 04:22 (sunday) = RESOLUTION: Fixed in e-smith-base-5.2.0-78.el5.sme * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme - Prevent weekly rebuilds of RAID arrays, by Mark Casey [SME: 6160] = CURRENT VERSION INSTALLED: [root@test81 ~]# rpm -qa e-smith-base e-smith-base-5.2.0-75.el5.sme =TESTING: Known issue, reports from prodbox: -------- Original Message -------- Subject: RebuildStarted event on /dev/md1:gateway0.burnat.com Date: 22 Jun 2013 18:22:01 -0000 From: mdadm monitoring <root@burnat.com> To: admin_raidreport@burnat.com This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildStarted event has been detected on md device /dev/md1. -------- Original Message -------- Subject: RebuildFinished event on /dev/md1:gateway0.burnat.com Date: 22 Jun 2013 18:22:04 -0000 From: mdadm monitoring <root@burnat.com> To: admin_raidreport@burnat.com This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildFinished event has been detected on md device /dev/md1. -------- Original Message -------- Subject: RebuildStarted event on /dev/md2:gateway0.burnat.com Date: 22 Jun 2013 18:22:03 -0000 From: mdadm monitoring <root@burnat.com> To: admin_raidreport@burnat.com This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildStarted event has been detected on md device /dev/md2. A Rebuild20 event has been detected on md device /dev/md2. A Rebuild40 event has been detected on md device /dev/md2. A Rebuild60 event has been detected on md device /dev/md2. A Rebuild80 event has been detected on md device /dev/md2. -------- Original Message -------- Subject: RebuildFinished event on /dev/md2:gateway0.burnat.com Date: 22 Jun 2013 20:39:23 -0000 From: mdadm monitoring <root@burnat.com> To: admin_raidreport@burnat.com This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildFinished event has been detected on md device /dev/md2. Device mismatches found: 512 is now an active member of md device /dev/md2. = UPDATED VERSION INSTALLED: [root@test81 ~]# yum --enablerepo=smetest update e-smith-base or if package available locally: [root@test81 files]# rpm -Uvh e-smith-base-5.2.0-78.el5.sme.i386.rpm [root@test81 files]# signal-event post-upgrade; signal-event reboot [root@test81 ~]# rpm -qa e-smith-base e-smith-base-5.2.0-78.el5.sme = PROBLEM FIXED?: a) Add a second drive to system -------- Original Message -------- Subject: DegradedArray event on /dev/md1:test81.schtrumpf.com Date: 25 Jun 2013 06:38:00 -0000 From: mdadm monitoring <root@schtrumpf.com> To: admin_raidreport@schtrumpf.com This is an automatically generated mail message from mdadm running on test81.schtrumpf.com. A DegradedArray event has been detected on md device /dev/md1. b)sync raid1 manually from console, emails received for each streps as usual: -------- Original Message -------- Subject: RebuildStarted event on /dev/md1:test81.schtrumpf.com Date: 25 Jun 2013 06:41:06 -0000 From: mdadm monitoring <root@schtrumpf.com> To: admin_raidreport@schtrumpf.com This is an automatically generated mail message from mdadm running on test81.schtrumpf.com. A RebuildStarted event has been detected on md device /dev/md1. - Device /dev/hdc1 is now an active member of md device /dev/md1. and: -------- Original Message -------- Subject: RebuildStarted event on /dev/md2:test81.schtrumpf.com Date: 25 Jun 2013 06:41:10 -0000 From: mdadm monitoring <root@schtrumpf.com> To: admin_raidreport@schtrumpf.com This is an automatically generated mail message from mdadm running on test81.schtrumpf.com. A RebuildStarted event has been detected on md device /dev/md2. - A Rebuild20 event has been detected on md device /dev/md2. - A Rebuild40 event has been detected on md device /dev/md2. - A Rebuild60 event has been detected on md device /dev/md2. - A Rebuild80 event has been detected on md device /dev/md2. - A RebuildFinished event has been detected on md device /dev/md2. We now have to wait for next Sunday 04:22 to see what takes place.
(In reply to Brian Read from comment #101) > Please confirm command to install this fix and I'll also test it. yum --enablerepo=smetest update e-smith-base You are looking at updating to version 5.2.0-78, not yet propagated in the repos, may take a few hours. 5.2.0-77 is there at the moment... I will post a tentative testing schedule as soon as I can updatew and ensure all is well, please wait.
(In reply to chris burnat from comment #100) > (In reply to Ian Wells from comment #99) > > Fixed in e-smith-base-5.2.0-78.el5.sme > > > > * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme > > - Prevent weekly rebuilds of RAID arrays, by Mark Casey [SME: 6160] > > > > > > I don't have a RAID in my test VMs so this has not yet been tested beyond > > simple compilation. Once it works then it needs cloning to SME 9 > > I will test later today and report any problems. We also need others to > test this fix to ensure that it meets expectation, the more test the better. > Who is available from the long list of CC? > Thanks. Please confirm command to install this fix and I'll also test it.
(In reply to Ian Wells from comment #99) > Fixed in e-smith-base-5.2.0-78.el5.sme > > * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme > - Prevent weekly rebuilds of RAID arrays, by Mark Casey [SME: 6160] > > > I don't have a RAID in my test VMs so this has not yet been tested beyond > simple compilation. Once it works then it needs cloning to SME 9 I will test later today and report any problems. We also need others to test this fix to ensure that it meets expectation, the more test the better. Who is available from the long list of CC? Thanks.
Fixed in e-smith-base-5.2.0-78.el5.sme * Mon Jun 24 2013 Ian Wells <esmith@wellsi.com> 5.2.0-78.sme - Prevent weekly rebuilds of RAID arrays, by Mark Casey [SME: 6160] I don't have a RAID in my test VMs so this has not yet been tested beyond simple compilation. Once it works then it needs cloning to SME 9
Created attachment 4008 [details] mdevent patch Added patch that will be committed soon.
e-smith-base provides both /var/service/raidmonitor/run & /sbin/e-smith/mdevent I would like the idea of changing the run script, but it was introduced with Bug 496 'Better mdadm reporting' and I'll copy here: >>> Currently mdadm only reports on the following events: Fail FailSpare DegradedArray Attached is a script that will send emails for the following events: RebuildStarted Rebuild20 Rebuild40 Rebuild60 Rebuild80 RebuildFinished Fail FailSpare SpareActive DegradedArray The included file includes the updated run file for raidmonitor and the script/template for the email. I've tried to make it easy for translation into other languages (something mdadm doesn't to at the moment). The reason for doing this besides the translations is that I like notification when the array is back working as it should be and not just the failure events. >>> So Shad introduced this to improve the reporting and that is still valid today. ==> Just change mdevent. I'll commit the change in Comment #84 to CVS so that it appears in CVS soon.
(In reply to Mark Casey from comment #95) >Sorry again if this is a step backwards...I do agree my patch (with credit to >Filippo) would solve the immediate issue if this isn't the time for a larger >clean-up. As you can see, this bug is waiting on A Snr. Dev for guidance, your proposed changes will only delay the process further down the track. I propose to stick to original proposal agreed by Charlie, we can refine later if not sufficient. # Restating request for help from a Snr Dev to facilitate resolution and patch making: Assistance from a Senior Dev required about how to deal with mods to template: Template associated with mdadm-2.6.9-5.el5 a) Try to template 99-raid-check, i.e. have the template system overwrite that file b) add the template to another package, i.e. e-smith-base owns the template c) for us to build the package and maintain a patch - cleaner but more work. Thanks
(In reply to chris burnat from comment #93) > (In reply to Charlie Brady from comment #92) > > (In reply to chris burnat from comment #91) > > > Having another look at this issue: > > > > > > ...truncated... > > > > > > > Looks good to me. > > Assistance from a Senior Dev required about how to deal with mods to > template: > > Template associated with mdadm-2.6.9-5.el5 > a) Try to template 99-raid-check, i.e. have the template system overwrite > that file > > b) add the template to another package, i.e. e-smith-base owns the template > > c) for us to build the package and maintain a patch - cleaner but more work. > > Thanks I have (1.) a possible correction there, and then (2.) I also have a question for a Senior Dev. 1.) Unless I'm misunderstanding the intent of your post, I think you're wanting to target '/sbin/e-smith/mdevent' for templating and not '/etc/cron.weekly/99-raid-check'. The proposed patches here were against mdevent, which is actually hopefully easier since I'm assuming it belongs to a package wholly "owned" by SME and is outside of the mdadm package. (You'll have to forgive me...this second part got long. Essentially my question is: "Why can't we just remove '/sbin/e-smith/mdevent' and let mdadm handle it?" I think the patch we're looking at here would fix the immediate problem but the more I look at it, it feels like a temporary band-aid. Anyway...) 2.) I wanted to mention/point out that the mdevent script is not only not a part of the mdadm package...it is completely optional! mdadm is fully capable of sending the email notices on its own, the mdevent handler script is just a customization (I'm assuming it was added to change how the notices look or which types are sent[?]). SME's raidmonitor service is currently called as: '/sbin/mdadm --monitor --scan --program /sbin/e-smith/mdevent'. So to my question for a Dev...I just tried stopping raidmonitor on one of my SME8 hosts and then I temporarily set the email option in /etc/mdadm.conf and started my own 'mdadm --monitor' process *without* the '--program /sbin/e-smith/mdevent' flag. I just let it sit in the foreground of my SSH session and block and do its thing. At that point, from another window, I made a few invocations to 99-raid-check after excluding our really large array. I also got ballsy and failed a member device out of /boot and then re-added it a few times and let it re-sync. The net effect was that mdadm on its own behaved almost identical to the mdevent handler. The curious thing was that it didn't send any of the Rebuild[20,40,60...] status notices. I'd never seen that, so now I'm wondering if mdadm is customized too... The point I'm trying to come around to is that I'm wondering if "our" mdadm was at some point customized to send or not send certain notices, and then that wasn't enough so mdevent was added, and now that isn't doing exactly what we want so we're gonna patch it, and then...I mean are we just adding more layers of junk? Sorry again if this is a step backwards...I do agree my patch (with credit to Filippo) would solve the immediate issue if this isn't the time for a larger clean-up.
(In reply to chris burnat from comment #93) > (In reply to Charlie Brady from comment #92) > > (In reply to chris burnat from comment #91) > > > Having another look at this issue: > > > > > > a) Taking into account Shad's comment #51 and Charlie comment #77, it may be > > > that Mark's variant in comment #84 is preferable: > > > > > > [/sbin/e-smith/mdevent] > > > print "Event: $event, Device: $device, Member: $member\n"; > > > > > > +if ($event =~ m#^Rebuild# && system( "ps -C 99-raid-check" ) == 0 ) { > > > + exit 0; > > > +} > > > + > > > if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { > > > my $domain = $conf->get_value("DomainName") || 'localhost'; > > > my $user = "admin_raidreport\@$domain"; > > > > Looks good to me. > > Assistance from a Senior Dev required about how to deal with mods to > template: > > > Template associated with mdadm-2.6.9-5.el5 > a) Try to template 99-raid-check, i.e. have the template system overwrite > that file > > b) add the template to another package, i.e. e-smith-base owns the template > > c) for us to build the package and maintain a patch - cleaner but more work. > > Thanks Could we finish this one ASAP and include it in the next ISO release you all are working on? Please. Thanks.
(In reply to Charlie Brady from comment #92) > (In reply to chris burnat from comment #91) > > Having another look at this issue: > > > > a) Taking into account Shad's comment #51 and Charlie comment #77, it may be > > that Mark's variant in comment #84 is preferable: > > > > [/sbin/e-smith/mdevent] > > print "Event: $event, Device: $device, Member: $member\n"; > > > > +if ($event =~ m#^Rebuild# && system( "ps -C 99-raid-check" ) == 0 ) { > > + exit 0; > > +} > > + > > if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { > > my $domain = $conf->get_value("DomainName") || 'localhost'; > > my $user = "admin_raidreport\@$domain"; > > Looks good to me. Assistance from a Senior Dev required about how to deal with mods to template: Template associated with mdadm-2.6.9-5.el5 a) Try to template 99-raid-check, i.e. have the template system overwrite that file b) add the template to another package, i.e. e-smith-base owns the template c) for us to build the package and maintain a patch - cleaner but more work. Thanks
(In reply to chris burnat from comment #91) > Having another look at this issue: > > a) Taking into account Shad's comment #51 and Charlie comment #77, it may be > that Mark's variant in comment #84 is preferable: > > [/sbin/e-smith/mdevent] > print "Event: $event, Device: $device, Member: $member\n"; > > +if ($event =~ m#^Rebuild# && system( "ps -C 99-raid-check" ) == 0 ) { > + exit 0; > +} > + > if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { > my $domain = $conf->get_value("DomainName") || 'localhost'; > my $user = "admin_raidreport\@$domain"; Looks good to me. > b) The next step is to schedule the "check" at a different time so as not to > conflict with clamscan. How to achive this, relocate 99-raid-check > somewhere else (where?) and making a separate entry in etc/crontab for a > different time weekly? IMO that should be a separate bug (perhaps NFR).
Having another look at this issue: a) Taking into account Shad's comment #51 and Charlie comment #77, it may be that Mark's variant in comment #84 is preferable: [/sbin/e-smith/mdevent] print "Event: $event, Device: $device, Member: $member\n"; +if ($event =~ m#^Rebuild# && system( "ps -C 99-raid-check" ) == 0 ) { + exit 0; +} + if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { my $domain = $conf->get_value("DomainName") || 'localhost'; my $user = "admin_raidreport\@$domain"; b) The next step is to schedule the "check" at a different time so as not to conflict with clamscan. How to achive this, relocate 99-raid-check somewhere else (where?) and making a separate entry in etc/crontab for a different time weekly? Comments please.
I propose to use Filippo's changes, and will proceed with it if no objections received over next few days. Thanks.
(In reply to David Bray from comment #87) > > You seem to have just as good a check doing a straight > cat /sys/block/md2/md/mismatch_cnt > > anything other than zero send yourself a message That is exactly what it does as-provided in mdadm, except that it sleeps in the background to wait to tally the counts, can be configured to skip devices, and will not issue a check to a device if it is already checking. As Filippo mentioned mdadm should only use idle i/o, but if you're still seeing a hit there or perhaps to the cpu you could reduce '/proc/sys/dev/raid/speed_limit_max' on a per-system basis. As you may know that parameter is just a goal, so you'd want to check /proc/mdstat during a check to get a better idea of what the current rate actually is before reducing the limit.
> Does that patch address anything other than suppress the message ? Yes. > The server performance will still severely degrade - on a raid5 with 4 x 1 > terrabyte drives - it takes several hours to recover. Performance shouldn't degrade too much, because disk bandwidth is consumed only if idle. See this /var/log/messages excerpt: May 5 04:22:02 mail kernel: md: syncing RAID array md2 May 5 04:22:02 mail kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc. May 5 04:22:02 mail kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction. System load avg goes slightly over 1.
Hi Does that patch address anything other than suppress the message ? The server performance will still severely degrade - on a raid5 with 4 x 1 terrabyte drives - it takes several hours to recover. It's not really a sme issue, I'd like to have the check, but have resorted to disabling it - the patch doesn't seem to encourage re-enabling it. You seem to have just as good a check doing a straight cat /sys/block/md2/md/mismatch_cnt anything other than zero send yourself a message
Thank you Filippo, 500 servers is a convincing test bed! We should solve this issue before releasing an new ISO for version 8. I propose to make the changes, commit in CVS and build the package for verification and release. Would one of the Senior Dev consider this proposal please?
> Filippo, just found your patch: > > -if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { > +if ($event =~ m#^Fail|^Degraded|^SpareActive#) { > > Is it still current, please confirm, than I will test it on a couple of > boxes I have available. The patch is used in production on around 500 server since last february: * mar feb 12 2013 Giacomo Sanchietti <giacomo@nethesis.it> 5.2.0-65-3nh - Do not send alert for Rebuild raid event. Fix [SME: 6160] We didn't experience many disk failures, but mdadm events were signalled by email.
(In reply to chris burnat from comment #83) > (In reply to Filippo Carletti from comment #71) > > > The bug is with raid1. If you are seeing a rebuild event on raid5 then it > > > is a legit rebuild because blocks were out of sync. > > > > Checking an array will always generate at least two emails, regardless of > > array type (RAID1 or RAID5). > > If the 99-raid-check command finds a mismatch_cnt != 0 on a raid5 array it > > will send an additional email just to say it found problems. > > > > I think that's what we want. > > Anybody tested my patch? > > > > Notes: > > 1. A mismatch_cnt different from 0 may mean problems, but the purpose of the > > weekly script is "to scrub" the array, and scrubbing is obtained with the > > check command. > > > > 2. Generating an email only for Fail|Degraded|SpareActive events alerts the > > sysadmin of a real problem. > > Filippo, just found your patch: > > -if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { > +if ($event =~ m#^Fail|^Degraded|^SpareActive#) { > > Is it still current, please confirm, than I will test it on a couple of > boxes I have available. > Thanks > chris Here is another variant that works: ====================================== @@ -37,6 +37,10 @@ print "Event: $event, Device: $device, Member: $member\n"; +if ($event =~ m#^Rebuild# && system( "ps -C 99-raid-check" ) == 0 ) { + exit 0; +} + if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { my $domain = $conf->get_value("DomainName") || 'localhost'; my $user = "admin_raidreport\@$domain"; ====================================== The only difference being that you'll still get notices about progress of any legitimate Rebuilds (i.e.: User physically replaced a bad disk, and gets emails on rebuild progress) on the other ~26 days of the month when 99-raid-check isn't running. I prefer personally this version, but I concede that as far as I know "Fail", "Degraded", and "SpareActive" by themselves are sufficient in any given situation to alert an admin to log in and check things out.
(In reply to Filippo Carletti from comment #71) > > The bug is with raid1. If you are seeing a rebuild event on raid5 then it > > is a legit rebuild because blocks were out of sync. > > Checking an array will always generate at least two emails, regardless of > array type (RAID1 or RAID5). > If the 99-raid-check command finds a mismatch_cnt != 0 on a raid5 array it > will send an additional email just to say it found problems. > > I think that's what we want. > Anybody tested my patch? > > Notes: > 1. A mismatch_cnt different from 0 may mean problems, but the purpose of the > weekly script is "to scrub" the array, and scrubbing is obtained with the > check command. > > 2. Generating an email only for Fail|Degraded|SpareActive events alerts the > sysadmin of a real problem. Filippo, just found your patch: -if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { +if ($event =~ m#^Fail|^Degraded|^SpareActive#) { Is it still current, please confirm, than I will test it on a couple of boxes I have available. Thanks chris
(In reply to Randy Williams from comment #81) > (In reply to chris burnat from comment #80) > > (In reply to Randy Williams from comment #79) > > > > > This bug is becoming troublesome and holding back some upgrades I want to do > > > so I am pleased to see this progress. > > > > Troublesome it is, even frustrating, agreed. However, how can it hold back > > "some" upgrades? Please elaborate. > > Thanks. > > I currently have a production server that takes over 26 hrs to complete a > raid rebuild. I do not want other servers to experience the slowness that > ensues as a result of this condition, which they will as they are similarly > configured. Thanks Randy, makes sense! Would one for the senior Dev provide guidance please, this problem needs some form of official fix I think...
(In reply to chris burnat from comment #80) > (In reply to Randy Williams from comment #79) > > > This bug is becoming troublesome and holding back some upgrades I want to do > > so I am pleased to see this progress. > > Troublesome it is, even frustrating, agreed. However, how can it hold back > "some" upgrades? Please elaborate. > Thanks. I currently have a production server that takes over 26 hrs to complete a raid rebuild. I do not want other servers to experience the slowness that ensues as a result of this condition, which they will as they are similarly configured.
(In reply to Randy Williams from comment #79) > This bug is becoming troublesome and holding back some upgrades I want to do > so I am pleased to see this progress. Troublesome it is, even frustrating, agreed. However, how can it hold back "some" upgrades? Please elaborate. Thanks.
(In reply to dakruhm from comment #78) > Created attachment 3833 [details] > raid-check from mdadm-3.2.3-9 > > >>Alternatively we could figure out how to run the check without generating a complaint about mismatch_cnt. We would also want to run it at a different time than clamscan. Doing the check is useful. > > If this is fixed in a newer version, out of curiosity, what would happen if > we backported v3.2.3-9 or re-packaged it with the newer raid-check script? > > It seems to lead us to either ignore or suppress the messages and move the > check to another time. This bug is becoming troublesome and holding back some upgrades I want to do so I am pleased to see this progress. Thank you dakruhm. Am I right to assume that your attached script is to replace 99-raid-check in cron.weekly? I don't want to implement it in the wrong way. Please confirm that to implement this fix, all I need do is replace 99-raid-check with your attached script. thank you.
Created attachment 3833 [details] raid-check from mdadm-3.2.3-9 >>Alternatively we could figure out how to run the check without generating a complaint about mismatch_cnt. We would also want to run it at a different time than clamscan. Doing the check is useful. If this is fixed in a newer version, out of curiosity, what would happen if we backported v3.2.3-9 or re-packaged it with the newer raid-check script? It seems to lead us to either ignore or suppress the messages and move the check to another time.
(In reply to dakruhm from comment #76) > In theory, you should be able to fix it by (replace X for number 1,2,etc): > #echo repair >/sys/block/mdX/md/sync_action > #echo check >/sys/block/mdX/md/sync_action Well, no - the point made by https://bugzilla.redhat.com/show_bug.cgi?id=547128 is that in theory and in practice mismatch_cnt being non-zero can occur without error when RAID1 is used (for swap) and doesn't need to be repaired, and doing those actions won't "repair" the non-problem. > So, if you have a raid1 (and not everyone does) and if the system is having > issues with mismatch_cnt (and not every raid1 system does), it's best to > turn it off here: > /etc/sysconfig/raid-check > > You want to do this so that the "check" which is often mistaken for a > "repair" (aka rebuild) doesn't happen every week at 4:22 which takes an > unusually long time because it coincides with clamscan and can cause the > system to become unresponsive. > > Correct? Pretty much. Alternatively we could figure out how to run the check without generating a complaint about mismatch_cnt. We would also want to run it at a different time than clamscan. Doing the check is useful. We want to detect hard drive issues sooner rather than later.
So, you should be able to see a history here: #cat /var/log/raidmonitor/current You can see if there's a current mis-match here: #cat /sys/block/md*/md/mismatch_cnt In theory, you should be able to fix it by (replace X for number 1,2,etc): #echo repair >/sys/block/mdX/md/sync_action #echo check >/sys/block/mdX/md/sync_action The explanation of why it doesn't work here: https://bugzilla.redhat.com/show_bug.cgi?id=547128 It looks like it fixed around here: mdadm-3.1.2-10 (or possibly here: mdadm-3.1.4-1) But we are here: # rpm -q mdadm mdadm-2.6.9-5.el5 So, if you have a raid1 (and not everyone does) and if the system is having issues with mismatch_cnt (and not every raid1 system does), it's best to turn it off here: /etc/sysconfig/raid-check You want to do this so that the "check" which is often mistaken for a "repair" (aka rebuild) doesn't happen every week at 4:22 which takes an unusually long time because it coincides with clamscan and can cause the system to become unresponsive. Correct?
Hi, I have two Sme-servers, one has RAID 1 for md1 and md2, the other has RAID 5 for md2 (3 disks, no spare). Since january-updates, every Sunday at 4.22 i get mails : Server 1 - RAID 1 (md1) + RAID 1 (md2) : md1 : RebuildStart event and rebuild finished event for md1 md2 : RebuildStart event and a real rebuild for md2 : Rebuild20 .... -> RebuildFinished event at a later time Server 2 - RAID 1 (md1) + RAID 5 (md2) : md1 : RebuildStart event and rebuild finished event for md1 md2 : RebuildStart event and a real rebuild for md2 : Rebuild20 .... -> RebuildFinished event at a later time Before the january-updates at 4.22 i got mails : Server 1 - RAID 1 (md1) + RAID 1 (md2) : md1 : RebuildStart event and rebuild finished event for md1 SOMETIMES : md2 : RebuildStart event and a real rebuild for md2 : Rebuild20 .... -> RebuildFinished event at a later time Server 2 - RAID 1 (md1) + RAID 5 (md2) : md1 : RebuildStart event and rebuild finished event for md1 So, no mails about md2/RAID 5
I have 2 boxes running SME 8.0 with all updates as of this date. The boxes are set as RAID1 for md1 and md2. I am experiencing the same problem as mentioned here. The Sunday rebuild on one box lasted 26 hrs! I have read though the comments and am a little confused. It seems a rebuild event is performed on a raid1 partition even if it is not needed. What I would like to know is how this is to be fixed. I see the tests for raid5, but I do not have raid5 so I don't see how that fix would work. I am only an admin not a dev so I need more explanation. If I can help with tests, please instruct. Thanks. PS the only contrib on the boxes is SME8ADMIN.
Created attachment 3739 [details] obsoleted patch to 99-raid-check; progress tracking with raidmonitor stop/restart
Ah, okay I missed your patch before. Up until now I'd thought mdadm itself was sending these emails; I'd seen the --program flag in the process listing but somehow it hadn't clicked for me that the --program was the one sending them. As it happens, 99-raid-check sleeps in the background while the checks are happening so it can tally the results when they're done. If your patch could source /etc/sysconfig/raid-check and see which devices are supposed to be checked, any Rebuild alerts for those devices could be dropped any time that 99-raid-check is found to be running. For what its worth, I'd written an addition to 99-raid-check which monitors each raid volume's check completion via /proc/mdstat and stops raidmonitor/mdadm -only-around the times that a useless email would go out (it works but needed much further testing). I think a patch to /sbin/e-smith/mdevent is the better way to go here, but I'm attaching the 99-raid-check patch anyway in case someone finds it searching and can reuse the parsing stuff at some point (and, really, so I feel better about the time spent). :D
> The bug is with raid1. If you are seeing a rebuild event on raid5 then it > is a legit rebuild because blocks were out of sync. Checking an array will always generate at least two emails, regardless of array type (RAID1 or RAID5). If the 99-raid-check command finds a mismatch_cnt != 0 on a raid5 array it will send an additional email just to say it found problems. I think that's what we want. Anybody tested my patch? Notes: 1. A mismatch_cnt different from 0 may mean problems, but the purpose of the weekly script is "to scrub" the array, and scrubbing is obtained with the check command. 2. Generating an email only for Fail|Degraded|SpareActive events alerts the sysadmin of a real problem.
(In reply to john judd from comment #69) > Comment on attachment 3729 [details] > dummy Raid 5 command prompt output > > I received 4 emails: > 1-A DegradedArray event has been detected on md device /dev/md39. > 2-A SpareActive event has been detected on md device /dev/md39. > Device /dev/loop3 is now an active member of md device /dev/md39 > 3-A RebuildStarted event has been detected on md device /dev/md39 > 4-A RebuildFinished event has been detected on md device /dev/md39. Thanks for checking that. With the disk images at only 30mb I thought it would happen too fast to send you those first two, but they are not a problem... from man madam: > When creating a RAID5 array, mdadm will automatically create a > degraded array with an extra spare drive. This is because building > the spare into a degraded array is in general faster than resyncing the > parity on a non-degraded, but not clean, array. This feature can be > overridden with the --force option. So depending on hard drive speed and the size of the disk image files created with 'dd', a person testing this may or may not see those the DegradedArray and SpareActive messages.
Comment on attachment 3729 [details] dummy Raid 5 command prompt output I received 4 emails: 1-A DegradedArray event has been detected on md device /dev/md39. 2-A SpareActive event has been detected on md device /dev/md39. Device /dev/loop3 is now an active member of md device /dev/md39 3-A RebuildStarted event has been detected on md device /dev/md39 4-A RebuildFinished event has been detected on md device /dev/md39.
Created attachment 3729 [details] dummy Raid 5 command prompt output
(In reply to john judd from comment #66) > Mark, I just ran your commands and I can confirm that my md2 mismatch count > is 0 but running echo "check" > /sys/block/md2/md/sync_action initiates the > rebuild of the raid 5, see comment 59 for previous confirmation of this > behaviour. Thanks for progressing this, John Well, yes. I was kind of vague calling it a device that was "confirmed-clean". I don't personally have any doubts that your md2 raid5 is clean, but it could well be hundreds of gb and have been in service for years so I thought it could still be speculated on in the end. I figured that creating a small brand new raid5 device like this would be further assurance that there hasn't even been time for something bad to happen to it, having just been initially synced. Also, a lot of people do not have a raid5 on their system so this makes it very easy to test. You mentioned you ran the commands; did you get a rebuild email for md39 when checking it?
Mark, I just ran your commands and I can confirm that my md2 mismatch count is 0 but running echo "check" > /sys/block/md2/md/sync_action initiates the rebuild of the raid 5, see comment 59 for previous confirmation of this behaviour. Thanks for progressing this, John
Could a few people try the following on an SME8 *DEV* box? (It didn't hurt my prod box but that would be at your own risk! :) I'd like to confirm I'm not the only one who also has this issue on confirmed-clean non-raid1 devices. The following will set up a "dummy" raid5, check it like 99-raid-check does, and then remove it. You should get mail even though the raid5 is healthy. This assumes /dev/md39 is not an active device on your box. # Create work folder mkdir /root/b6160raid_images # Create disk images dd if=/dev/zero of=/root/b6160raid_images/disk1 bs=1 count=0 seek=30M dd if=/dev/zero of=/root/b6160raid_images/disk2 bs=1 count=0 seek=30M dd if=/dev/zero of=/root/b6160raid_images/disk3 bs=1 count=0 seek=30M # Make available on loop devices losetup /dev/loop1 /root/b6160raid_images/disk1 losetup /dev/loop2 /root/b6160raid_images/disk2 losetup /dev/loop3 /root/b6160raid_images/disk3 # Set up "dummy" raid5 mdadm --create /dev/md39 --level=5 --raid-devices=3 --metadata=0.90 /dev/loop1 /dev/loop2 /dev/loop3 #### You must wait about 1 minute for the initial sync to finish, as shown in 'cat /proc/mdstat'. #### # Confirm that mismatch_cnt is 0 cat /sys/block/md39/md/mismatch_cnt # Run a check and get alerts on a perfectly healthy raid5 echo "check" > /sys/block/md39/md/sync_action When you're done: # Clean up mdadm --stop /dev/md39 losetup -d /dev/loop1 losetup -d /dev/loop2 losetup -d /dev/loop3 rm /root/b6160raid_images/disk{1..3} rmdir /root/b6160raid_images/
Could a few people try this on a dev box (it shouldn't hurt )?
(In reply to Bob Stalzer from comment #62) > That is what I am seeing. Three weeks in a row. > Because of that I am reluctant to do an upgrade > > Bob Thanks Bob. Here the changelogs for mdadm: * Thu Apr 05 2012 Jes Sorensen <Jes.Sorensen@redhat.com> - 2.6.9-5 - Update the raid-check script to run checks on arrays that are clean and either 'idle' or 'active' as opposed to just 'idle'. Otherwise checks will often be missed on arrays that are regularly busy. - Resolves: bz735803 * Sat Mar 31 2012 Jes Sorensen <Jes.Sorensen@redhat.com> - 2.6.9-4 - Fix /sbin/raid-check to not report non-0 mismatch counts on raid1 arrays, as these can happen legitimately on raid1 arrays. - Fix build problem due to CVS Revision ID change in mdmpd due to switch from CVS to git. - Resolves: bz566828 * Fri Nov 13 2009 Doug Ledford <dledford@redhat.com> - 2.6.9-3 - If we have more than one array per drive, then the raid-check script will only check one of the arrays, the others will get skipped. Correct that. - Resolves: bz523000 I checked my main server and the latest package was updated on Jan 21 18:46:32 Updated: mdadm-2.6.9-5.el5.i386. I think I was a little late updating, version 2.6.9-5 appears in smeupdates in date of the 10 Jan 2013.
(In reply to chris burnat from comment #61) > (In reply to Bob Stalzer from comment #60) > > FWIW,I have 2 servers running sme8. One has all but the last updates, the > > other is one step back. the one that needs the most updates has > > mdadm-2.6.9-3.el5. It is reporting the md1 rebuild at 4:22 every Sunday. The > > more update server has mdadm-2.6.9-5.el5, and it doing the md1 rebuild and > > has been doing the md2 rebuild also. > > Bob, > Can you confirm that the server with the older mdadm-2.6.9-3.el5 is only > rebuilding md1 whilst the server with the newer mdadm-2.6.9-5.el5 (latest > package) is rebuilding both md1 AND md2? > > This would be significant I think. > Thanks That is what I am seeing. Three weeks in a row. Because of that I am reluctant to do an upgrade Bob
(In reply to Bob Stalzer from comment #60) > FWIW,I have 2 servers running sme8. One has all but the last updates, the > other is one step back. the one that needs the most updates has > mdadm-2.6.9-3.el5. It is reporting the md1 rebuild at 4:22 every Sunday. The > more update server has mdadm-2.6.9-5.el5, and it doing the md1 rebuild and > has been doing the md2 rebuild also. Bob, Can you confirm that the server with the older mdadm-2.6.9-3.el5 is only rebuilding md1 whilst the server with the newer mdadm-2.6.9-5.el5 (latest package) is rebuilding both md1 AND md2? This would be significant I think. Thanks
FWIW,I have 2 servers running sme8. One has all but the last updates, the other is one step back. the one that needs the most updates has mdadm-2.6.9-3.el5. It is reporting the md1 rebuild at 4:22 every Sunday. The more update server has mdadm-2.6.9-5.el5, and it doing the md1 rebuild and has been doing the md2 rebuild also.
Ref. Mark's comment 55, my servers are completely unmodified from stock SME&contribs. Pre 25th Jan I received weekly RebuildStarted/RebuildFinished event messages regarding device /dev/md1 only, both of which were timestamped at 04:22am. Post the update I still receive the md1 messages but now receive additional messages regarding md2, resulting in weekly rebuilds of the Raid 5 md2. So something changed in the update, it seems to me that md2 wasn't being checked previously.
Created attachment 3694 [details] tests on CentOS 6.3
I forgot to mention before that on my SME8, when echoing "check" to md1, if I run `service raidmonitor stop`, then mdadm --monitor disappears from ps -ef and the emails stop. With raidmonitor/mdadm started back up the emails come back when running check. Anyway I tried a few things in a CentOS 6.3 VM installed from the minimal iso to see how the newer stuff handles things (the console log is attached as MC-md-tests-cent6_3.txt). I set up a raid1 and a raid5 on sparse loop devices and set up mdadm's email notifications, then ran a check on the arrays. I did not get emails when running check operations like I did on my SME8 box. Then I tried writing from dd/urandom to one "disk" in each array to corrupt it. Even though mismatch_cnt did start to go up from my corrupting data, I never got a single email from mdadm (and I guess that's not terrible as long as raid-check is around, though I'm now more surprised it excludes raid1 and 10 across the board). I then went fairly crazy on it with dd, enough so that it couldn't rebuild when I --failed a device and tried to remove/add it back, and it still showed mismatch_cnt at 4096 after a repair and check. Still, no emails about check or repair or corruption. (the only exception was I did get mail for the aforementioned --fail I ran, which shows notifications were working) In the end I'm pretty sure the difference is simply that this CentOS box is running mdadm v3.2.3 and my SME8 box is running mdadm v2.6.9. I looked for a few minutes and I didn't see anything in mdadm's release notes, but they probably just turned off the check notice emails as redundant since things like raid-check seem common now. I'm pretty sure I read it was new in Cent 5.
I hope it isn't sacrilege to say this here...but on one of my Ubuntu boxes... :) The contents of the file /etc/cron.d/mdadm is: # By default, run at 00:57 on every Sunday, but do nothing unless the day of # the month is less than or equal to 7. Thus, only run on the first Sunday of # each month. crontab(5) sucks, unfortunately, in this regard; therefore this # hack (see #380425). 57 0 * * 0 root [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ] && /usr/share/mdadm/checkarray --cron --all --quiet Now, I'm assuming this is a Debian-ism or Ubuntu-ism, but the part that interests me is that the checkarray command has a --quiet flag. That flag results in no emails from mdadm --monitor, but your kernel log still gets: > [386328.911187] md: data-check of RAID array md0 > [386328.911192] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > [386328.911194] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > [386328.911200] md: using 128k window, over a total of 1465138304 blocks. > [412279.013549] md: md0: data-check done. > [412279.127741] RAID5 conf printout: > [412279.127748] --- rd:6 wd:6 > [412279.127751] disk 0, o:1, dev:sda > [412279.127752] disk 1, o:1, dev:sdb > [412279.127754] disk 2, o:1, dev:sdc > [412279.127755] disk 3, o:1, dev:sdd > [412279.127757] disk 4, o:1, dev:sde > [412279.127758] disk 5, o:1, dev:sdf Of course THEN the question is: does it give a warning like 99-raid-check does if it actually DOES find a problem. I've been meaning to set up some arrays on loop devices and try some things on that box and on my SME8 host but I've actually got 99-raid-check disabled on SME and our fileserver no longer uses md for production purposes, so I haven't been able to find/justify time to mess with it in depth. Actually the thing to do is probably to set up a dummy raid1 and raid5 on an el6 system and throw some echo "check" commands at them and see what happens.
(In reply to Shad L. Lords from comment #53) > (In reply to john judd from comment #52) > > some of my comments from duplicate bug 7247 were copied into comment 33: > > "2.6.18-348.1.1.el5. Previously the 'rebuild' only affected md1 but this > > week it's additionally rebuilding md2." > > My md1 is raid 1 but md2 is Raid 5. Pre 25th Jan the problem only affected > > md1 but post the update it started affecting md2 which is raid 5: > > The bug is with raid1. If you are seeing a rebuild event on raid5 then it > is a legit rebuild because blocks were out of sync. I believe that is incorrect. My /boot is the default raid1 on md1 and if I run echo "check" > /sys/block/md1/md/sync_action ...I will get RebuildEvent emails. It doesn't matter if I do it 9 times in a row, I will get 9 sets of emails (I am waiting for each rebuild to finish before re-running the command; not sure what it does if you don't wait). IMO these emails are not a report of a problem, they are a report that problems are being looked for. The report of the problem would come from 99-raid-check itself in the form of an email stating: "WARNING: mismatch_cnt is not 0 on /dev/md1". (Though it would not do that for a raid1 because the block of code I pasted in Comment 49 explicitly excludes them, but it would send it for his raid5 if there really was a problem) > > but post the update it started affecting md2 which is raid 5 This is only because previous to January's update for the mdadm package he had either customized the options in /etc/sysconfig/raid-check, or the defaults for the mdadm package (or perhaps the default-at-install /etc/sysconfig/raid-check) used to only check md1 and post-update the default ended up including a check for md2. CAVEAT: Hopefully this is obvious, but I don't want to mislead in saying I don't think John Judd has a problem here...: If you're getting RebuildEvent emails AND mismatch_cnt emails then you DO have a problem.
OK but is it possible that something changed to affect raid 5 check, was it not running correctly previously? I have 3 SME 8 servers, none of which rebuilt raid 5 pre 25th Jan, all of which now weekly rebuild at the same time as the md1 rebuild.
(In reply to john judd from comment #52) > some of my comments from duplicate bug 7247 were copied into comment 33: > "2.6.18-348.1.1.el5. Previously the 'rebuild' only affected md1 but this > week it's additionally rebuilding md2." > My md1 is raid 1 but md2 is Raid 5. Pre 25th Jan the problem only affected > md1 but post the update it started affecting md2 which is raid 5: The bug is with raid1. If you are seeing a rebuild event on raid5 then it is a legit rebuild because blocks were out of sync.
some of my comments from duplicate bug 7247 were copied into comment 33: "2.6.18-348.1.1.el5. Previously the 'rebuild' only affected md1 but this week it's additionally rebuilding md2." My md1 is raid 1 but md2 is Raid 5. Pre 25th Jan the problem only affected md1 but post the update it started affecting md2 which is raid 5: md2 : active raid5 sdd2[2] sdc2[0] sdb2[3](S) sda2[1] 1953310720 blocks level 5, 256k chunk, algorithm 2 [3/3] [UUU]
I wouldn't exclude rebuild events. If you are really rebuilding an array you want to know about it. I propose that we template the config and exclude any RAID1 arrays. They are the ones that have issues.
Oops, the system where I checked the 99-raid-check script was not up-to-date. mdadm-2.6.9-5.el5 has the script backported from el6. I propose to change our /sbin/e-smith/mdevent excluding Rebuild event notifications. -if ($event =~ m#^Rebuild|^Fail|^Degraded|^SpareActive#) { +if ($event =~ m#^Fail|^Degraded|^SpareActive#) {
(In reply to Ian Wells from comment #48) > (In reply to Filippo Carletti from comment #47) > > > The next step is to develop a good fix, Comment #42 could be one approach. > > > > Another option is to backport the script form el6. > > Could you say more, what has been changed for el6? Not a whole lot. I was wondering myself so I put up a VM of CentOS 6.3. Both it and SME have a section like the following text... (the file is /etc/cron.weekly/99-raid-check on my SME8 box, and /usr/sbin/raid-check on the new 6.3 VM) excerpt from SME version; note the comment: -------------------------------------------------------------------- for dev in $check_list; do mismatch_cnt=`cat /sys/block/$dev/md/mismatch_cnt` # Due to the fact that raid1 writes in the kernel are unbuffered, # a raid1 array can have non-0 mismatch counts even when the # array is healthy. These non-0 counts will only exist in # transient data areas where they don't pose a problem. However, # since we can't tell the difference between a non-0 count that # is just in transient data or a non-0 count that signifies a # real problem, simply don't check the mismatch_cnt on raid1 # devices as it's providing far too many false positives. But by # leaving the raid1 device in the check list and performing the # check, we still catch and correct any bad sectors there might # be in the device. raid_lvl=`cat /sys/block/$dev/md/level` if [ "$mismatch_cnt" -ne 0 -a "$raid_lvl" != "raid1" ]; then echo "WARNING: mismatch_cnt is not 0 on /dev/$dev" fi done ------------------------------------------------------------------- That section is very similar in the newer version on 6.3 too, which just expands the reporting exception at the end to cover raid10 too instead of just raid1. So the Rebuild event notices being discussed here aren't being sent via cron by /etc/cron.weekly/99-raid-check. They're from the process '/sbin/mdadm --monitor --scan --program /sbin/e-smith/mdevent'. That process is reporting that 99-raid-check requested the check be run around line 53. The emails are not a report that the check went badly, just that the check was requested and is being performed (which I agree is dumb, considering the script that requested it is about to ignore the results for raid1 volumes anyway). As that comment mentions though, the reason not to disable it altogether is that while running 'check' (as opposed to 'repair') on the raid1 will not attempt to fix data disagreements between two otherwise healthy sectors (because it has a 50/50 shot of screwing it up), it *will* still correct cases where one disk develops a bad sector and the other appears okay. Am I just repeating the obvious or was there previously not consensus on these points? This bug seems to come and go in popularity and it is hard to tell what the prevailing wisdom is, so forgive me if I've done the former. :) Also, mdadm was in that January update so it tracks that a lot of people are saying they had deleted raid-check and it came back, or that a different set of volumes started getting checked then.
(In reply to Filippo Carletti from comment #47) > > The next step is to develop a good fix, Comment #42 could be one approach. > > Another option is to backport the script form el6. Could you say more, what has been changed for el6?
> The next step is to develop a good fix, Comment #42 could be one approach. Another option is to backport the script form el6.
(In reply to Paul Floor from comment #45) > How many more comments are we going to require from users proving that > "raid-check" does not work properly on Raid-1 volumes before we decide to > release a patch to disable it on Raid-1 volumes? The evidence from users indicates that this could be handled better. So far there is no fix to release. The next step is to develop a good fix, Comment #42 could be one approach.
How many more comments are we going to require from users proving that "raid-check" does not work properly on Raid-1 volumes before we decide to release a patch to disable it on Raid-1 volumes?
Reporting same issue since installation of fresh install(s). Disabeling raid-check on both healthy systems until resolution is found. Nothing new to report that has not already been stated below. **note ** attempted to move to monthly check - no joy - no explination.
Reporting same issue, appearing over the last month or so, on a RAID 1 system. Running short SMART tests (smartctl -t short /dev/sda; smartctl -t short /dev/sdb) on the disks produce no evidence of problems. No errors have been reported in the SMART logs as well.
I suggest creating a template "/etc/e-smith/templates/etc/sysconfig/raid-check" that will generate /etc/sysconfig/raid-check content. if system is raid 1 : ENABLED=no if there is no raid : ENABLED=no if there is raid 5 only : ENABLED=yes if there is raid 5 and raid 1 : enabled only for raid 5 volumes : ENABLED=yes CHECK=check # To check devs /dev/md0 and /dev/md3, use "md0 md3" CHECK_DEVS="md2 md3" # raid 5 REPAIR_DEVS="" SKIP_DEVS="md1" # raid 1 possibility to override it with db configuration what I am not sure : - in what rpm include it - best way to check for raid and volume ( cat /proc/mdstats ?) - should add some link to the template to which signal-event ?
Ditto, Have now had rebuild event on md2 now occur over last month or so on Sunday. Prior to this always got the Sunday check of md1 only. Two SME 8 systems, all updates, my home, and another in a small NFP org. Both running Raid1 with 2 500gb drives all drives are less than 6 months old.
(In reply to purvis from comment #39) > I will be double checking ... > I will try and review .. > I am now setting up some other servers from older replaced equipment... That's great, please add any observations that you have as they can help us fix this.
On all SME 8.0Servers, since the upgrade in the later part of January 2013, I have Raid 1 running and no spares. I am seeing the raid rebuilds too. Previously to that update, I had removed the file "/etc/cron.weekly/99-raid-check" on all the servers. The file found it's way back on my server after the update. I will be deleting 99-raid-check on my raid 1 servers until there is a reason to have it running. I cannot see how all my servers would have a need to rebuild something that is not broke on raid 1 system. I will be double checking that smart is active on all my drives with the command "smartctl -s on -o on -S on /dev/sd[ab]" on all my bare metal servers. I will try and review the file "/var/log/raidmonitor/current" and see if there where any md2 rebuilds messages prior to that update. I usually delete the file "/var/log/raidmonitor/current file" after reviewing and solving any problems with raid 1 drives. I am now setting up some other servers from older replaced equipment. I will try to setup a new test server and hopefully can do not production testing there.
(In reply to Peter Jones from comment #37) > I went through the 4.22am scenario again today also. > > I am really suspicious there is more to this than meets the eye. For exactly > four weeks I have had an issue with getting emails hourly from CLAM telling > me it cannot download the daily patch file but when I run freshclam - v it > is telling me it is up to date. > > This issue started IMMEADIATELY after the 4.22am RAID scenario. > > Today, exactly 4 weeks later, (1 calendar month) this issue mysteriously > dissapeared IMMEADIATELY AFTER the 4.22am RAID scenario. > > Is this a coincidence or is there more to this ?? Please open a separate bug for your freshclam issue. Thanks
I went through the 4.22am scenario again today also. I am really suspicious there is more to this than meets the eye. For exactly four weeks I have had an issue with getting emails hourly from CLAM telling me it cannot download the daily patch file but when I run freshclam - v it is telling me it is up to date. This issue started IMMEADIATELY after the 4.22am RAID scenario. Today, exactly 4 weeks later, (1 calendar month) this issue mysteriously dissapeared IMMEADIATELY AFTER the 4.22am RAID scenario. Is this a coincidence or is there more to this ??
Raising to MAJOR
Created attachment 3620 [details] Temperature graph for array during md2 Rebuild Its Sunday Oz time again, and this time ALL of my version 8 servers went into a spin and performed a Rebuild of md2. Example: #START Date: 2 Feb 2013 17:44:04 -0000 This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A Rebuild20 event has been detected on md device /dev/md2. #END Date: 2 Feb 2013 19:39:29 -0000 This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildFinished event has been detected on md device /dev/md2. Device mismatches found: 256 is now an active member of md device /dev/md2. Note: other server did not have the Device mismatches shown above. Raising to MAJOR, not sure what effect it has on drives, cannot be beneficial to long term reliability. See temperature graph attached.
(In reply to chris burnat from comment #33) > (In reply to chris burnat from comment #32) > > *** Bug 7247 has been marked as a duplicate of this bug. *** > > Reprinting summary of the two posts in Bug #7247: > > Quack 2013-01-28 10:27:08 MST > > For the second Sunday in a row, I get emails from mdadm monitoring To: > admin_raidreport. I have received no other messages indicating disk errors. > ..... > 160 or so updates were applied on 25th Jan, incl kernel version > 2.6.18-348.1.1.el5. Previously the 'rebuild' only affected md1 but this week > it's additionally rebuilding md2. Yes, I am seeing that too, since updateing, it used to just re-build md1, last Sunday it re-built md2 as well.
(In reply to chris burnat from comment #32) > *** Bug 7247 has been marked as a duplicate of this bug. *** Reprinting summary of the two posts in Bug #7247: Quack 2013-01-28 10:27:08 MST For the second Sunday in a row, I get emails from mdadm monitoring To: admin_raidreport. I have received no other messages indicating disk errors. (REFERENCE SME 8.0 FORUM POST jAN 28 2013 02:44:32 AM "MDADM Monitoring Reports Rebuild Events Detected and SME Server Slows to a Crawl") Here are the messages I received today (in order received). A RebuildStarted event has been detected on md device /dev/md1. A Rebuild60 event has been detected on md device /dev/md1. A RebuildStarted event has been detected on md device /dev/md2. A RebuildFinished event has been detected on md device /dev/md1. A Rebuild20 event has been detected on md device /dev/md2. (5 hours after start) A Rebuild40 event has been detected on md device /dev/md2. (10+ hours after start) At about when the Rebuild20 message was received, 5 hours after the RebuildStart message, incoming emails stopped, and trying to access the server-manager through a browser and log in with SSH become essentially non-responsive. Maybe one out of 10 tries on either will provide a login prompt but won't display anything else. As a result, I cannot look at the mdadm log at the moment. Incoming emails are also delayed 30 minutes to over an hour. [snip, there is more info] and: I can confirm this behaviour on three SME 8 servers, my log files below: typical weekly log entry prior to 25th Jan yum update: 2013-01-20 04:22:02.113570500 Event: RebuildStarted, Device: /dev/md1, Member: 2013-01-20 04:22:04.273375500 Event: RebuildFinished, Device: /dev/md1, Member: log entry post 25th jan update: 2013-01-27 04:22:02.532572500 Event: RebuildStarted, Device: /dev/md1, Member: 2013-01-27 04:22:04.654465500 Event: RebuildStarted, Device: /dev/md2, Member: 2013-01-27 04:22:05.799639500 Event: RebuildFinished, Device: /dev/md1, Member: 2013-01-27 04:51:07.469983500 Event: Rebuild20, Device: /dev/md2, Member: 2013-01-27 05:19:08.977772500 Event: Rebuild40, Device: /dev/md2, Member: 2013-01-27 05:49:09.969894500 Event: Rebuild60, Device: /dev/md2, Member: 2013-01-27 06:23:11.060000500 Event: Rebuild80, Device: /dev/md2, Member: 2013-01-27 07:04:52.431831500 Event: RebuildFinished, Device: /dev/md2, Member: 160 or so updates were applied on 25th Jan, incl kernel version 2.6.18-348.1.1.el5. Previously the 'rebuild' only affected md1 but this week it's additionally rebuilding md2.
*** Bug 7247 has been marked as a duplicate of this bug. ***
Created attachment 3599 [details] kernel messages raid rebuild Logs attached showing activity during rebuild of array
Re-opening this bug, it may need a second look... I am myself experiencing similar events. As an example, on my production server fitted with 2 x 1TB Sata drives, SME8 32 bits system, I received every week at 04:22 local time (Sydney) the following notifications: [04:22] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildStarted event has been detected on md device /dev/md1. [04:22] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildFinished event has been detected on md device /dev/md1. However, this week, I received the following: [04:22] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildStarted event has been detected on md device /dev/md1. [04:22] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildStarted event has been detected on md device /dev/md2. [04:22] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildFinished event has been detected on md device /dev/md1. [04:45] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A Rebuild20 event has been detected on md device /dev/md2. [05:09] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A Rebuild40 event has been detected on md device /dev/md2. [05:34] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A Rebuild60 event has been detected on md device /dev/md2. [06:03] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A Rebuild80 event has been detected on md device /dev/md2. [06:40] This is an automatically generated mail message from mdadm running on gateway0.burnat.com. A RebuildFinished event has been detected on md device /dev/md2. Device mismatches found: 384 is now an active member of md device /dev/md2.
I suggest to write a template disabling this on raid 1 system as a fix ? I am not sure it's good to run a rebuild or even only a scan that last 3 to 4 hours on a disk for its life expectancy. Yes, I understand it is better to be sure of data consistency, but as far as I understood from all that has been reported here, it does not seem to be working correctly on RAID 1. Therefore, to me, it seems like it is producing the exact opposite to its design with RAID1 : highest risk of data loss...
What is the status of this ?? I see it flagged NOTABUG. I have been experiencing exactly this for the last few weeks on a ne Server. Sure as heck looks like it is doing a rebuild. Behavious exactly mirrors that described in comment #21
(In reply to Paul Floor from comment #26) > You should probably read comments 1 through 10. The Raid Check mechanism > does not work well on Raid 1, you should just disable it or it will most > likely (and unnecessarily) rebuild your arrays every week. if it were the only problem it should rebuild all the raid 1 devices,but it only rebuilds md1 (boot), md2 (swap) but not md3 (system and data) on a no lvm system.
You should probably read comments 1 through 10. The Raid Check mechanism does not work well on Raid 1, you should just disable it or it will most likely (and unnecessarily) rebuild your arrays every week. > Does the Paul Floor solution from comment 5 work? No, it made it worse. > Any other leads to try to get this problem resoved? Raid check doesn't seem to work well on Raid 1 arrays, I ended up turning this off on my Raid 1 servers by removing 99-raid-check from /etc/cron/weekly.
(In reply to Charlie Brady from comment #23) > It's not doing a rebuild, but it is doing a whole surface scan of the array, > which is what takes so long. Sorry to insist I was buying this explanation, but if it is only doing a scan that is detected as a rebuild : why i receive two emails for /dev/md0 ( boot) and two for /dev/md2 ( swap) but none for the system disk /dev/md3 every weeks? If the test should trigger the event as if it were a rebuild, so i should see an email for this partition too, shouldn't I ? Also, if i manually launch the script i have emails for the three + Rebuild20, Rebuild40 Rebuild60 Rebuild80 events in the following 2 hours for md3. based on that, i see three solutions : - raid is so bad that it make so many errors that we should test it every day or every hour ( or better avoid using it) - the script is not performing well - naming it 99-* is not enough to have it running last as other script in /etc/cron.weekly are not named with heading numbers, thus md3 is not idle when trying to access it and it is not tested ... and if i want only one array to be tested this is the one with my data, not the swap or the boot ... On the other hand i thought that cron did not start next script before the previous was not finished # cat /etc/sysconfig/raid-check #!/bin/bash # # Configuration file for /etc/cron.weekly/raid-check [...] # Note: the raid-check script intentionaly runs last in the cron.weekly # sequence. This is so we can wait for all the resync operations to complete # and then check the mismatch_count on each array without unduly delaying # other weekly cron jobs. If any arrays have a non-0 mismatch_count after # the check completes, we echo a warning to stdout which will then me emailed # to the admin as long as mails from cron jobs have not been redirected to # /dev/null. We do not wait for repair operations to complete as the # md stack will correct any mismatch_cnts automatically. [...] ENABLED=yes CHECK=check # To check devs /dev/md0 and /dev/md3, use "md0 md3" CHECK_DEVS="" REPAIR_DEVS="" SKIP_DEVS="" # ll /etc/cron.weekly/ total 20 -rwxr-xr-x 1 root root 380 mars 28 2007 0anacron -rwxr-xr-x 1 root root 1938 janv. 26 2010 99-raid-check -rwxr--r-- 1 root root 1376 août 23 2006 mailsorting -rwxr-xr-x 1 root root 414 mai 30 08:34 makewhatis.cron -rwxr--r-- 1 root root 58 nov. 20 2005 refresh-ddns
(In reply to Charlie Brady from comment #23) > It's not doing a rebuild, but it is doing a whole surface scan of the array, > which is what takes so long. should we document that on the wiki so the information will be available for everybody easily?
(In reply to Mark Casey from comment #21) > I do not think it is just sending the email, I think it is actually > rebuilding because the md1 email for /boot emails that it is finished > immediately but md2 does not send the finished email until much later in the > day (10-11 hours later). It's not doing a rebuild, but it is doing a whole surface scan of the array, which is what takes so long.
Just saw this text from the "See Also" link in this bug's header info (it is to another bug from RHEL that was dealing with roughly the same topics): ------------------------------------------------------------- This bug is closed because the original reporter was claiming that even though the script purported to be performing a check, he was convinced it was performing a repair instead. This isn't true, it's just that /proc/mdstat uses resync as the state whether it's doing a check or a repair. ------------------------------------------------------------- Before I saw that I was doing more testing on my own and came to the same conclusion (so I was wrong in my last comment). mismatch_cnt is not ever going above 0 here. I'm still not sure if everyone using md RAID1 gets the same Rebuild emails every week, but I am content that in spite of them we're currently not in a situation that actually causes a rebuild. ymmv We already had an upgrade planned that will make the issue go away so I'm just going to set the check job's config to skip the arrays in question.
Is this bug still ongoing (or could it be reopened?). I upgraded to SME8 recently and now for 3 weeks my raid has rebuilt every Sunday morning at 4:22am. I check the mismatch_cnt on both md1 and md2 a few days ago (Friday) and both were at 0. I do not think it is just sending the email, I think it is actually rebuilding because the md1 email for /boot emails that it is finished immediately but md2 does not send the finished email until much later in the day (10-11 hours later). I'm not getting any email about the mismatch_cnt, only the notices from mdadm. Not sure what to do next.
same problem here: i have both md1 and md2 on a sme nolvm version 8 installation since ist installation : from july the 1st at 4:22 everyweek. curiously it never happens on md3. could be avoid this 2 mails for each array if nothing is or document that on the wiki considering the bug #3535 ?
(In reply to Marco Hess from comment #17) > I then performed the repair action on md2 like in: > > # echo repair > /sys/block/md2/md/sync_action I was preparing this note while I was waiting for the result of the above command and accidentally pressed the Save Changes :-). Anyway, the interesting thing is that at the end of that first repair on md2 the system still reported differences in the RAID. I then ran the repair command again and then the number of mismatches became 0. In hindsight I might have run repair more than once on the md1 array as well while I was figuring out the command syntax and not being sure it actually ran. I am thinking that the /etc/cron.weekly/99-raid-check runs a 'check' action that counts the differences in the RAID but doesn't repair anything. Then supposedly the 'repair' action also goes through but actually repairs the differences but it may be that it counts the number of repairs it has done and hence at the end of the repair the mismatch count is still non 0. Then the second time there are really no difference anymore and the mismatch count finally returns to 0? I ran /etc/cron.weekly/99-raid-check and it kicked off a 'check' resync of both the md1 and md2 arrays and still checked out with 0 mismatches. Personally I find it a bit disheartening to see these RebuildStarted/Finished events emails all the time even though it appears to be just a check operation. Also, we might need to provide a way for a less tech savy admin to run these 'repair' events or at least document them as a potential solution to these check events that keep reporting non 0 mismatch counts and nothing seems to get fixed.
Should we REOPEN this based on comment #17?
My system also rebuilds both the md1 and md2 arrays every week (Sunday 4:22) and kept on having non zero mismatch counts (also shown in the RebuildFinished event messages) like in: | Device mismatches found: 128 is now an active member of md device /dev/md1. | Device mismatches found: 4992 is now an active member of md device /dev/md2. Looking for more info on the issue, I found the following which seems to make some sense: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=405919 So I tried: # echo repair > /sys/block/md1/md/sync_action After that a rebuild occurred and the mismatch count returned to 0. Also the RebuildFinished email message no longer has the mismatch message at the end: | This is an automatically generated mail message from mdadm running on mydomain.com. | A RebuildFinished event has been detected on md device /dev/md1. I then performed the repair action on md2 like in: # echo repair > /sys/block/md2/md/sync_action
Since the linked bug in RedHat's bugzilla is CLOSED NOTABUG as it is a check and not a rebuild, I wonder if we should keep this open.
*** Bug 6562 has been marked as a duplicate of this bug. ***
*** Bug 5669 has been marked as a duplicate of this bug. ***
To paul floor: you can also add the (raid1) arrays that do you not want to be checked to appropriate parameters in /etc/sysconfig/raid-check, or turn it of by setting ENABLED to no.
Last nights Cron log shows no log evidence, both awstats and fetchmail are installed, these are the 5 min messages. It execute cron.weekly and 4:22 (the anacron update message proves that) which executes the 99-raid-check script: Oct 10 04:15:01 server02 crond[11508]: (root) CMD (/etc/startmail) Oct 10 04:15:01 server02 crond[11509]: (root) CMD (/bin/nice /sbin/e-smith/awstats-pp -s -n) Oct 10 04:20:01 server02 crond[11554]: (root) CMD (/bin/nice /sbin/e-smith/awstats-pp -s -n) Oct 10 04:20:01 server02 crond[11555]: (root) CMD (/etc/startmail) Oct 10 04:20:01 server02 crond[11558]: (root) CMD (/usr/lib/sa/sa1 1 1) Oct 10 04:22:01 server02 crond[11591]: (root) CMD (run-parts /etc/cron.weekly) Oct 10 04:22:01 server02 anacron[11595]: Updated timestamp for job `cron.weekly' to 2010-10-10 Oct 10 04:25:01 server02 crond[11720]: (root) CMD (/etc/startmail) Oct 10 04:25:01 server02 crond[11721]: (root) CMD (/bin/nice /sbin/e-smith/awstats-pp -s -n) Oct 10 04:30:01 server02 crond[11797]: (root) CMD (/bin/nice /sbin/e-smith/awstats-pp -s -n) Oct 10 04:30:01 server02 crond[11798]: (root) CMD (/usr/lib/sa/sa1 1 1) The mail that is sent: from: mdadm monitoring [root@mydomain.org] to: admin_raidreport@mydomain.org This is an automatically generated mail message from mdadm running on server02.mydomain.org RebuildStarted event has been detected on md device /dev/md2. I searched al files in /var/log that have been changed in the past 24 hours for relevant messages but I cannot find any related log info. Mdadm log only showed the rebuild event messages. I investigated the 99-raid-check script. This script will check for each md# the array_state to be clean and status to be idle. If so, it echoes the word "check" in the file /sys/block/$dev/md/sync_action. After that the script waits until the state returns to idle. When all devices have been done, it will check the mismatch_cnt files for values ne 0. If found it prints "WARNING: mismatch_cnt is not 0 on /dev/$dev" I tried to run the 99-raid-check script manually twice. The first no rebuilds (the expected behaviour), the second time it started rebuilding both the md1 and md2 array. -> The check action itself is handled by another proces which performs the actual rebuild. -> The answer to why it is rebuilding can be answered by investigating the proces (the mdadm monitor?) Since my server is a VM on an VMware Esxi machine, I can monitor the VM's disks, the rebuild is not a "real rebuild" since the system only reads from both disks (hi-speed 65MB/s). The write rate is very low. If I login as admin during a rebuild, the raid state monitor (option 5) shows a resync action is active. These are the facts for now. Any leads to what's next?
(In reply to comment #8) > Created attachment 3111 [details] > messages log kernel messages which show rebuilds This doesn't show why the rebuild has started. One or both of you will need to find more information, in cron mail messages and/or messages log, to indicate why the rebuild was started.
(In reply to comment #9) > Does the Paul Floor solution from comment 5 work? No, it made it worse. > Any other leads to try to get this problem resoved? Raid check doesn't seem to work well on Raid 1 arrays, I ended up turning this off on my Raid 1 servers by removing 99-raid-check from /etc/cron/weekly.
After the smeb6 update, 6 weeks ago, my server started rebuilding md2. This is a 1TB partition and takes about 5 hours to complete. I have had this problem with the md1 partition since I installed the server (sme8b3). The server runs as a VM on a Esx4.2i host, two separate disks on separate VMware storages. Installed default (raid 1, two disks, no spare). output of cat /proc/mdstat Personalities : [raid1] md1 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU] md2 : active raid1 sdb2[1] sda2[0] 1073631936 blocks [2/2] [UU] unused devices: <none> I attached the relevant kernel messages from the messages log for the last to rebuild events (attachment 3111 [details]) Does the Paul Floor solution from comment 5 work? Any other leads to try to get this problem resoved?
Created attachment 3111 [details] messages log kernel messages which show rebuilds
(In reply to comment #6) > Here's a clean description of the problem with our version: > http://rhn.redhat.com/errata/RHBA-2010-0006.html I'm fairly sure that is a different issue. I think we are seeing this: http://www.fractalizer.ru/frpost_350/warning-mismatch_cnt-is-not-0-on-devmd0/
(In reply to comment #0) > Every Sunday morning for the last 3 weeks at 4:22 I recieve 2 emails stating > that my raid is rebuilding and then the rebuild is complete. These are about weekly scrubbing of RAID arrays. See bug 3535 for details. Here's a clean description of the problem with our version: http://rhn.redhat.com/errata/RHBA-2010-0006.html
As an experiment I have adjusted my crontab via custom templates. 3:00 Backup (usually completes within 30 minutes) 4:02 check4updates 5:02 cron.daily (with raid-check moved to /etc/cron.daily and renamed zzz-99-raid-check so it runs dead last). This will cause it to run every night instead of once a week. Let's see if this makes any difference.
(In reply to comment #3) > It is very possible if you drive does 25MB/s throughput then it would only take > 4 seconds to rebuild a 100MB slice. That does make perfect sense as they are 15K SCSI drives. > EL5 added a new script (/etc/cron.weekly/99-raid-check) that checks raid arrays > for consistency and rebuilds them if they are consistent. Did you mean "rebuilds them if they are *NOT* consistent"? As far as I can tell, this script is supposed to check the array's consistency and only repair bad sectors. If it finds good sectors that contain bad data, it will increments a counter file. It appears that my md1 device is not in sync when the check is performed. I did see this little tidbit... # Note: the raid-check script intentionaly runs last in the cron.weekly # sequence. This is so we can wait for all the resync operations to complete # and then check the mismatch_count on each array without unduly delaying # other weekly cron jobs. If any arrays have a non-0 mismatch_count after # the check completes, we echo a warning to stdout which will then me emailed # to the admin as long as mails from cron jobs have not been redirected to # /dev/null. We do not wait for repair operations to complete as the # md stack will correct any mismatch_cnts automatically. We are not running this as the last cron job as intended because we have jobs not starting with numbers... 0anacron 99-raid-check makewhatis.cron refresh-ddns I don't know that it matters except that this script runs at the same time as check4updates. Some posts indicate this can be problematic if write operations are occuring during the check, especially on raid1 arrays. If yum is reading/writing to the drive when raid-check runs, this may be causing the array to appear inconsitent and force a repair. 22 4 * * 0 root run-parts /etc/cron.weekly 22 4 * * 0 root /sbin/e-smith/check4updates -m
> The weekly raid rebuild takes 4 seconds. That partition is only 100MB (25MB > used) but is it really doing a rebuild in 4 seconds? It is very possible if you drive does 25MB/s throughput then it would only take 4 seconds to rebuild a 100MB slice. > I guess the real question is why does it rebuild every week at the same time. > I never get any warning emails telling me that md1 is out of sync and have been > watching the raid set and I have never seen it out of sync. EL5 added a new script (/etc/cron.weekly/99-raid-check) that checks raid arrays for consistency and rebuilds them if they are consistent. I remember reading some posts about there being some issues with it and this is most likely the case. I don't remember if there was ever a solution or resolution to this issue though. Might want to search the redhat/centos mailing list archives and see if you can come up with anything. I'll do the same and post back anything I find.
(In reply to comment #1) > (In reply to comment #1) > > Also originally set this up as 2 disk mirror plus spare. Is this normal to > > only have the spare on md2? If not normal, is it related or a separate issue? > > > > [root@mail ~]# cat /proc/mdstat > > Personalities : [raid1] > > md1 : active raid1 sdc1[2] sdb1[1] sda1[0] > > 104320 blocks [3/3] [UUU] > > > > md2 : active raid1 sdc2[2](S) sdb2[1] sda2[0] > > 35800768 blocks [2/2] [UU] > Yes this is correct. md1 is the boot partition and grub, md1 should be active > and running on all drives in the system. With md1 being raid1 you can have all > but 1 drive fail and still have it work. It doesn't hurt to have the raid1 > active on all drives. > md2 creates a spare so that it can recover if one of the drives fails. Thanks for the clarification. The weekly raid rebuild takes 4 seconds. That partition is only 100MB (25MB used) but is it really doing a rebuild in 4 seconds? I guess the real question is why does it rebuild every week at the same time. I never get any warning emails telling me that md1 is out of sync and have been watching the raid set and I have never seen it out of sync.
(In reply to comment #1) > Also originally set this up as 2 disk mirror plus spare. Is this normal to > only have the spare on md2? If not normal, is it related or a separate issue? > > [root@mail ~]# cat /proc/mdstat > Personalities : [raid1] > md1 : active raid1 sdc1[2] sdb1[1] sda1[0] > 104320 blocks [3/3] [UUU] > > md2 : active raid1 sdc2[2](S) sdb2[1] sda2[0] > 35800768 blocks [2/2] [UU] Yes this is correct. md1 is the boot partition and grub, md1 should be active and running on all drives in the system. With md1 being raid1 you can have all but 1 drive fail and still have it work. It doesn't hurt to have the raid1 active on all drives. md2 creates a spare so that it can recover if one of the drives fails.