[SOLVED] - logrotate monthly at midnight a custom file

[warron.french]

We have a file called /var/log/ETA.log

We need to analyze all of the data in it once per month and therefore have only the last month's data in it. In other words, today is 01MAY2020 and I want to have only May's data in the logfile so that in June when I inspect/analyze/evaluate the data I can be certain that it ONLY HAS dates for May in it.

So, my coworker setup a logrotate policy in /etc/logrotate.d, called ETA:
The policy syntax looks like the following:

Code: Select all

/var/log/ETA.log {
        rotate 3
        missingok
        monthly
        dateyesterday
        delaycompress
        compress
        copytruncate
}

He implemented the policy /etc/logrotate.d/ETA into action, on 16APR2020, by doing the following:

Code: Select all

logrotate -f /etc/logrotate.d/ETA

Then last night the logfile, /var/log/ETA.log, rolled and compressed.

Code: Select all

NOTE: The syntax above has been updated today with "dateyesterday" and "delaycompress" because I want last month's date on the file instead, and not to compress the file until the next month over.

The problem is that the data in it includes data from today, 01MAY2020 all the way through "Fri May 1 03:15:01 EDT 2020"

We just want, for example, data from 01JUN2020 00:00:00 through 31JUN2020 23:59:59, to be in the rolled logfile for June, since we cannot fix it for May already. friberibitz!

I don't know how to force the logrotation of my specific file, to occur on the first day at 00:00:00 each month. Any ideas would be greatly welcomed, and maybe if they involve the /etc/anacrontab with some adjustments that would be fine; even if they don't they would be welcomed also.

[TrevorH]

Logrotate runs out of /etc/cron.daily at a random time controlled by parameters in /etc/anacrontab. By default that includes things like

RANDOM_DELAY=45
START_HOURS_RANGE=3-22

which means the jobs won't run until at least 03:00 with up to 45 minutes delay.

[warron.french]

TrevorH, correct, that's why I am asking to find out how to run /etc/logrotate.d/ETA immediately at 23:59:00 or 00:00:00.

Any ideas?

[pjsr2]

There are two ways of configuring jobs that you want to schedule regularly: through crontab or through anacron.

Jobs in /etc/cron.{daily,weekly,monthly} are handled by anacron. They are run approximately at the interval indicated in the name. The "approximately" part is controlled by three things: RANDOM_DELAY, START_HOURS_RANGE and whether your compter is running or not. Jobs are not started outside the START_HOURS_RANGE. When started, there is a RANDOM_DELAY (in minutes), which is to prevent that multiple jobs start at the same time and compete for resources. When your computer is not running/sleeping/hibernating at the scheduled time, it will be done at the first occasion when your computer is running/awake again (with some delay). This makes that scheduled tasks may be performed later, but they will be run at some time eventually. Extremely useful property for a backup job of a laptop!

Crontab jobs are executed at exactly the time you schedule them. See man crontab how to use is. The disadvantage is that when the computer is off/sleeping/hibernated at the scheduled time, the task will be skipped and never run.

You can use crontab to schedule a task at 00:00:00 of the first day of the month. Note that any task needs some time, so even when you schedule the job at 00:00:00 there still may be some log messages of the new month in the rotated log files.

[warron.french]

@psjr thanks,
Before I saw your reply today, I did a little experimentation with the logrotate policy and command running inside my crontab.

So in the /var/spool/cron/root (Root's crontab) I used this syntax starting off:

Code: Select all

0  0  *  *  *  /sbin/logrotate -f /root/logrot.stamptest

I didn't even know if this would work, but it got me very close to what I actually wanted; with the exception of data from the next day (by a few seconds) into the rolled logfile.

So I altered my Root's crontab to this state:

Code: Select all

59 23 * * * /sbin/logrotate -f /root/logrot.stamptest

This result for a daily rotation is exactly what I am looking; however, to take this to the next level is a little tricky for me so far because if I want to roll the data at 23:59hours on the last day of the current month I need to figure something out.

Is there something I can do to make this easier in the third column (day of month [1-31])? Is there a variable or something that I can use?

[warron.french]

There are two ways of configuring jobs that you want to schedule regularly: through crontab or through anacron.

Jobs in /etc/cron.{daily,weekly,monthly} are handled by anacron. They are run approximately at the interval indicated in the name. The "approximately" part is controlled by three things: RANDOM_DELAY, START_HOURS_RANGE and whether your compter is running or not. Jobs are not started outside the START_HOURS_RANGE. When started, there is a RANDOM_DELAY (in minutes), which is to prevent that multiple jobs start at the same time and compete for resources. When your computer is not running/sleeping/hibernating at the scheduled time, it will be done at the first occasion when your computer is running/awake again (with some delay). This makes that scheduled tasks may be performed later, but they will be run at some time eventually. Extremely useful property for a backup job of a laptop!

Crontab jobs are executed at exactly the time you schedule them. See man crontab how to use is. The disadvantage is that when the computer is off/sleeping/hibernated at the scheduled time, the task will be skipped and never run.

You can use crontab to schedule a task at 00:00:00 of the first day of the month. Note that any task needs some time, so even when you schedule the job at 00:00:00 there still may be some log messages of the new month in the rotated log files.

Also psjr2, as an added note can I put these two variables (RANDOM_DELAY, START_HOURS_RANGE) in my logrotate policy and alter their values for that specific policy?

[pjsr2]

RANDOM_DELAY and START_HOURS_RANGE are a property of anacron, not from your logrotate policy.
They are set in /etc/anacrontab.

The tasks in /etc/cron.{daily,weekly,monthly} are performed sequentially, not in parallel. The tasks are executed in the alphabetical order they appear in in the directory.

[TrevorH]

And the RANDOM_DELAY and START_HOURS_RANGE determine when it kicks off all the tasks in /etc/cron.daily so you could just adjust those to start the task with no delay and no range.

However... whatever method you use to do this, you are never going to be able to guarantee that the file will not contain extra records or be missing some. Tasks take time to run and other things can run while they're doing it so there is a window, albeit small, where your logrotate might start and new data might be written to the old day's file or perhaps data to be written might be delayed until after the file has been rotated so you might get yesterday's date in today's file. You would do better to adapt whatever processes this data to ignore data outside the ranges you want to process.

[pjsr2]

Crontab does not provide a special method to run a job at the last day of the month. So if you can live with starting at midnight, 00:00:00 of the first day is easier:

Code: Select all

0 0 1 * * myjob.sh

If you really want at the end of the day, you can do something like:

Code: Select all

59 23 30 4,6,9,11        * myjob.sh
59 23 31 1,3,5,7,8,10,12 * myjob.sh
59 23 28 2               * myjob.sh

but that is not correct in leap years, where you want to have Feb. 29th.

Something like

Code: Select all

59 23 28-31 * * [[ "$(date --date=tomorrow +\%d)" == "01" ]] && myjob.sh

could also do the trick. (Note escaping the % in the command, as % is special in crontab.) But for clarity, I would run it at 00:00:00 of the first of the month. The processing will take some time anyhow, so you will end up with some log messages from the day before or the next day. If that is unacceptable, you have to do special processing of the log data to filter it.

[warron.french]

@TrevorH and psjr2, thanks.

So I cannot use those variables in a standard crontab.

Can I use them in a custom anacron execution of logrotate somehow?