and APFS performance is known to deteriorate over time on HDD.
So I have manually explicitly enabled automatic defragmentation of the volume, though my reading suggests that this will not help much because metadata is the problem and is not defragmented. Also, I think that some automatic defrag was happening anyway, accounting for mysterious continuing activity when the drive should otherwise have been idle.
% diskutil apfs defragment disk5 status APFS Container defragmentation is currently disabled for disk5 % diskutil apfs defragment disk5 enable APFS Container defragmentation has been successfully enabled for disk5 % sudo diskutil apfs defragment disk5s2 status APFS Volume defragmentation is currently disabled for disk5s2 % sudo diskutil apfs defragment disk5s2 enable APFS Volume defragmentation has been successfully enabled for disk5s2 % sudo diskutil apfs defragment disk5s1 status APFS Volume defragmentation is currently disabled for disk5s1 % sudo diskutil apfs defragment disk5s1 enable APFS Volume defragmentation has been successfully enabled for disk5s1
As of I think that backups may be starting more quickly after I connect the external USB (HDD) drive.
I now look out for iostat
activity on disk4
falling to zero to indicate defrag is done, since the 'busy' indicator against the drive in Finder is now permanently on.
% iostat 5 disk0 disk4 cpu load average KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m 13.29 86 1.11 16.63 0 0.01 6 4 90 2.05 2.02 2.01 16.00 1 0.02 8.77 193 1.65 16 10 75 2.05 2.02 2.01 177.13 9 1.59 8.37 207 1.69 14 8 78 2.28 2.07 2.03 21.00 54 1.11 6.51 197 1.25 11 7 81 2.18 2.05 2.02 6.36 15 0.09 6.73 223 1.47 17 10 73 2.25 2.06 2.03 ... disk0 disk4 cpu load average KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m 13.24 428 5.53 0.00 0 0.00 5 3 91 1.22 1.23 1.45 10.37 10 0.10 0.00 0 0.00 2 2 96 1.36 1.26 1.46 14.62 76 1.08 13.96 18 0.25 4 3 93 1.33 1.26 1.46 28.13 6 0.17 0.00 0 0.00 4 2 94 1.31 1.25 1.46 5.54 35 0.19 0.00 0 0.00 3 2 95 1.28 1.25 1.45 0.00 0 0.00 0.00 0 0.00 2 2 96 1.34 1.26 1.46
The disk4
activity sometimes continues after the drive is ejected, which is somewhat alarming, though I assume safe else it would not happen.
For reference, some stats during an active backup, and entering the Cleaning Up...
phase:
11.79 60 0.69 5.88 115 0.66 5 4 91 1.93 1.93 1.98 7.42 98 0.71 5.16 117 0.59 10 8 82 1.93 1.93 1.98 4.16 76 0.31 8.94 78 0.68 5 5 90 1.78 1.90 1.96 5.35 80 0.42 8.51 99 0.82 7 7 87 1.80 1.90 1.96 7.12 60 0.42 7.18 76 0.53 8 6 86 1.89 1.92 1.97 9.42 908 8.36 9.62 75 0.71 11 8 81 1.90 1.92 1.97 345.37 179 60.24 282.00 150 41.26 5 6 90 1.83 1.90 1.96 216.09 343 72.42 483.07 175 82.69 7 6 87 2.24 1.99 1.99 354.43 302 104.65 940.50 100 92.02 3 4 92 2.06 1.95 1.98 94.76 97 8.97 202.23 95 18.70 7 5 88 2.14 1.97 1.99 13.40 608 7.96 10.63 96 1.00 7 6 87 2.05 1.95 1.98 7.79 37 0.28 4.34 156 0.66 5 5 90 2.20 1.99 1.99 55.91 66 3.59 4.77 96 0.45 7 4 89 2.10 1.97 1.99
After quite a long time of being told that old URLs are dead, many spiders are still trying to download from them.
To try to trim nugatory traffic a little, I have added some new rules to robots.txt
to deter access to large chunks of namespace currently empty and unsupported. (Some of this namespace may never be supported again.)
User-agent: * Disallow: /_cat/ Disallow: /_site/ Disallow: /_tn/ Disallow: /_virtual/
I have adjusted the configuration of a few more of the static sites (including m.earth.org.uk
) to actively filter out any If-None-Match
request headers to go along with no longer generating any ETag
. This will help ensure that conditional GET
s can work properly, even if clients retain and re-present very old ETag
values.
# Until Apache 2.5, disable ETag entirely for this site. FileETag none Header unset ETag +RequestHeader unset If-None-Match
I have added very minimal support for BibTex crossref
support, simply showing the citation token, not attempting to link, fill in fields, etc. I may not be using it as intended this way, but this can be improved.
I have adjusted P/p and Q/q to for a threshold CoP of 2, as that is the case now and shows that it can happen!
I have adjusted P/p to use CoP% (ie if current intensity is a little under predicted then the value is a little over 100), and over 150 now flags significant saving from topping-up storage.
This numeric value allows a smooth fading in of maximum top-up percentage from (say) a CoP% of 150 to 250, eg with a mean of 200%.
Right now I am seeing both P
and Q
flags, but no 'supergreen' G
(though grid is ordinary green).
2024-09-05T14:26Z MT 41 GI 78 IT 169 F PQbvx ES 1
A matching change has been made for calculating Q
/q
and a CoP >150% significance threshold.
The grid went supergreen at 4pm ie peak time (H
), so a top-up would have been locked out here:
2024-09-05T14:31Z MT 41 GI 78 IT 168 F PQbvx ES 1 2024-09-05T14:36Z MT 41 GI 78 IT 168 F PQbvx ES 1 2024-09-05T14:41Z MT 1 GI 78 IT 0 F PQbvx ES 1 2024-09-05T14:46Z MT 1 GI 78 IT 0 F PQbvx ES 1 2024-09-05T14:51Z MT 1 GI 80 IT 0 F PQbvx ES 1 2024-09-05T14:56Z MT 1 GI 82 IT 0 F PQbvx ES 1 2024-09-05T15:01Z MT 0 GI 86 IT 0 F HGPQbvx ES 1 2024-09-05T15:06Z MT 0 GI 85 IT 0 F HGPQbvx ES 1
(Intensity has sharply fallen over 24h and is due to rise again over 48h...)
Moments that are supergreen (G
) and with at least one of P
and Q
can be picked out with something like:
% egrep ' F [^ ]*[PQ]' data/heatBattery/log/live/20240905.log | egrep ' F [^ ]*G'
I added a Q
(and converse q
) flag (indicating that a big grid intensity fall has happened over the last 7d, so storing heat now would have an effective CoP >1) to script/heat-battery-target.sh
output to estimate how often it might drive a top-up, with an initial threshold value of a CoP >2.0.
Yes, all the good and obvious flag letters have been used...
I added a P
(and converse p
) flag (indicating that a big grid intensity rise is predicted over the next 48h, so storing heat now would have an effective CoP >1) to script/heat-battery-target.sh
output to estimate how often it might drive a top-up eg 2024-09-03T14:31Z MT 0 GI 225 IT 139 F Rpbvx ES 3
, with an initial threshold value of a CoP >2.0.
Google Search Console now reports that 29% of recent crawls of EOU by Google get 304s now. A number that has been rising steadily, mainly because I stopped using ETag
s I suspect.
Anticipting the heat pump getting installed, and thus gas consumption dropping to zero, I adjusted the recent utilities consumption working from the form At 16WW over 32 days from 2024-08-01 to 2024-09-02: gas consumption was 0.2kWh/d, electricity imports were 0.4kWh/d (net -9.5kWh/d).
to At 16WW over 32 days from 2024-08-01 to 2024-09-02: electricity imports 0.4kWh/d (net -9.5kWh/d), gas consumption 0.2kWh/d
, with the gas clause disappearing entirely when zero.