Wednesday 17 November 2010

Create task groups by TTY comment

I've had every man and his dog either drop into IRC or email me asking me what my thoughts are on the grouping tasks by tty layer patch discussed here: Phoronix link and slashdot. I guess people didn't understand my 2.6.36-ck1 announcement clearly enough, so I'll quote it again here:

Those following the development of the patches for interactivity at massive
load, I have COMPLETELY DROPPED them as they introduce regressions at normal
workloads, and I cannot under any circumstances approve changes to improve
behaviour at ridiculous workloads which affect regular ones. I still see
precisely zero point at optimising for absurd workloads. Proving how many
un-niced jobs you can throw at your kernel compiles is not a measure of one's
prowess. It is just a mindless test. 

Remember, I already had developed a hierarchical tree-based penalty patch for BFS and blogged about it here. I can do it in a 10 line patch for BFS, but it introduced regressions, which is why I dropped it (see earlier blog entry here: further-updates-on-hierarchical-tree).

Again, I can't for the life of me see why you'd optimise for make -j64 on a quad core machine. It is one workload, unique to people who compile all the time, but done in a way you wouldn't normally do it anyway. It is not going to magically make anything else better. If for some god-forsaken reason you wanted to do that, you could already do that with nice, or even better, by running it SCHED_IDLEPRIO.

nice -19 make -j 64 blahblah
or
schedtool -D -e make -j64 blahblah

It's not really that hard folks...

And if you really really really still want the feature for BFS, the patch that does the hierarchical tree based penalty is rolled into a bigger patch (so a lot more than just the 10 lines I mentioned) that can also group threads and enable/disable the features and it's still here:

bfs357-penalise_fork_depth_account_threads.patch

It is worth noting also that the mainline approach costs you in throughput, whereas this patch is virtually free.

EDIT: I forgot to mention that for YEARS now I've been using my "toolsched" wrapper scripts that do this automatically. See toolsched for the scripts. Make always starts as SCHED_IDLEPRIO for me at home.

41 comments:

  1. You might be true, but this patch still be good for desktop "normal" users, that have no idea of what tty, nice, task and kernel is. They like to open browsers with a lot of tabs, with a lot of Adobe Flash Player and other plugins, while listening to music and using some chat program.

    ReplyDelete
    Replies
    1. If somebody is still reading here:
      Could this kernel option
      /proc/sys/kernel/sched_autogroup_enabled
      result in a performance decrease of CPU intensive jobs and benchmarks like super-pi by about 30%?
      I observe this on a Core i7-980 with a 2.6.37 kernel, when comparing with a Core i7 960 with older kernel.

      " How to become a hacker Learn hacking step by step"

      Delete
  2. See that's the thing, it won't. The rest of all those normal desktop GUI apps won't be grouped. If you -do- group all firefox threads together as one app, or all amarok threads together as one app, both of those will perform -worse- under load.

    ReplyDelete
  3. Éimpressão minha ou tem um certo ar de inveja no ar?

    ReplyDelete
  4. Sorry, Portuguese is not one of the languages I speak. I've understood the words as "Impress me or a certain air of envy?". I don't want to guess your intent with this question without really understanding your language.

    ReplyDelete
    Replies
    1. Don't bother, it is syntactically incorrect in his own language.

      Delete
  5. So basically this patch is for developers that want to make -j64 and browse the net at the same time? If so, does it even deserve to be in the kernel if it doesn't do any good for a regular user?

    ReplyDelete
  6. I appreciate that developers tend to have a different view on what performance enhancements are necessary, but I wish that they could see things from the perspective of regular desktop users who almost never compile anything.

    I've tested the patch and it without a doubt improves the responsiveness of the desktop including Compiz. Most noticeably being if I have several applications running at once as I usually do. Firefox, gimp, inkscape, openoffice & synaptic updating is enough to make the desktop sputter, but this offers much improvement from what I can tell.

    ReplyDelete
  7. The Anonymous said "It's my impression or there's a certain air of envy in the air", like if you were feeling envy :P Weird...

    Anyway, I'm worried if that new kernel patch will make things worse under normal load. I read some comments saying that apps that weren't opened from a tty will get slower, or there will be no gains of performance at all...

    ReplyDelete
  8. Funny that whenever Con comes up with some scheduler idea (whether good or bad), the kernel guys will come up with a copy cat implementation a week after.

    ReplyDelete
  9. I read what you are saying, but for some reason with your version of this patch, I 'seem' to have complete control of my mouse cursor when many tabs are open in firefox, playing a flash movie, compiling a program in the background (not -j64 !), etc. With just the .357 patch, these activities occasionally seem to overtake access to the cpu. I understand that using nice or schedtool for compiling, but what is going on with these other activities? Is it just a faulty test on my part?

    Galen

    ReplyDelete
  10. Oh that patch DID make a difference (it's more than just tty groups). It's just that gnome applets sometimes failed to initialise, and mplayer stuttered under load.

    ReplyDelete
  11. About developers and desktop users. It's also a problem when the users does not listen to the developer.

    In this case a developer says it's a bad idea but users who show no indication of having any knowledge or experience about the kernel or kernel development thinks the patch should 'be used' because at the moment their desktops feel snappier, and that the developer should listen to their opinion. That is a very disrespectful behaviour by the user.

    ReplyDelete
  12. @Anonymous

    Quite frankly, a regular desktop user doesn't care about what a developer thinks or about the behind the scenes politics. They care about how their system operates and if they can have a UI that is more responsive with the same hardware, obviously they will choose the more responsive option.

    ReplyDelete
  13. That makes sense... but i just use it a couple hours which is decreasing my performance instead increase it

    ReplyDelete
  14. Hi Con,

    Lennart Poettering seems to agree with you that "make -j" is not valid desktop use case (http://lkml.org/lkml/2010/11/16/392).

    He also out-performed the new "miracle" patch with 4 lines of bash on vanilla kernel (http://lkml.org/lkml/2010/11/16/330), in response to a (rudely worded) challenge laid down by Linus (http://lkml.org/lkml/2010/11/16/298).

    I guess at least it's good that this issue is finally being acknowledged and taken seriously. Thanks for all your great work in this area (Kororaa Linux used -ck, back in the day).

    Cheers,
    Chris

    ReplyDelete
  15. I want the kind of responsiveness that your patch enables, more than I want gnome or mplayer. ;)

    Galen

    ReplyDelete
  16. Dear Brazilian friend: Don't do that, it's not polite and pretty childish. Take that from another Brazilian. Relaxa.

    As an user, I praise all 'easy' and transparent things, like patch -> build -> reboot and use.

    ReplyDelete
  17. Hi Con,

    nice link from Chris Smart to the discussion thread on the kml. Fine to see, that even the kernel programmers recognized the bad behavior for linux on some situations ;-) That's why I switched from cfs to bfs. And I'm happy with it. As a normal desktop user (he, what's that ;) ) I only need a quick reaction of my user input. Can't understand, why on modern computers this shouldn't be possible. And it's fine, that with BFS this does work. Simple and good. And if there is a simple algorithm to achieve this, than this is a working solution for almost all situations. It's nearly impossible (btw. seems that the scheduler of solaris has something similar, as Con wrote, but its a huge piece of software) to check all conditions and make a complicated model to support all situations. And not to break with the actual programs. As I mentioned already on the blog, would be fine if there is a automatic nice setting tool for programs, so that lazy users don't need to set it all. And now I see, that Con has something similar already with toolsched. Thanks Con.
    Conclusion: I confirm Cons solution.

    CU sysitos

    ReplyDelete
  18. so, the problem is "what is actually doing one type of user or another?" somebody should take that statistically, if it hasn't been done yet, sampling and so on!

    ReplyDelete
  19. @Con, wrong answer:
    You should have let it go without intervention. Then in the next cycle you could have shown up with your far more general solution and BFS would have been included into mainline :)

    ReplyDelete
  20. We added this patch to our kernel in Chakra Linux and right now we've only good reports about it, even more can be disabled adding a single word to the kernel line during the bootup, just test and use the feature if you need it. As i'm a packager for me works lke "awesome".

    ReplyDelete
  21. ...
    chakra-devel said...

    We added this patch to our kernel in Chakra Linux and right now we've only good reports about it, even more can be disabled adding a single word to the kernel line during the bootup, just test and use the feature if you need it. As i'm a packager for me works lke "awesome".

    20 November 2010 11:47 PM
    ...

    Which patch, con's fork-penalty patch?

    Galen

    ReplyDelete
  22. First I apologize for having to assert myself a tool like Google Translate to maybe be able to make myself understood to all who pass by your blog!
    I totally disapprove use of the anonymity to shouting insults, irony or to force a person look ridiculous or something, which seems to have been the case.
    I want to convey to you that many of us, simple users who have tried to upgrade our linux systems by testing patches and modifications, either by curiosity or need, using too your experience, your knowledge and even yourr mistakes, because only do not make mistakes those who do not try.
    So I would like to take this opportunity to also thanking you for all you has shared with us, inviting you, if you so wish, to participate collaboratively with your aknowledge in this topic (http://ubuntuforum-pt.org/index. php? topic = 29799.msg421708 # msg421708) whose only difficulty is to be in Portuguese.
    The topic was created by this user (http://ubuntuforum-pt.org/index.php?action=profile u = 18 026) which since 2007 has sought to share their knowledge with anyone related to compile a linux kernel (specifically ubuntu distro) that would provide a balance between performance and stability.
    Thank you for your generous attention and I apologize for the long post!

    ReplyDelete
  23. First I want to apologize for having to assert myself a tool like Google Translate to maybe be able to make myself understood to all who pass by your blog!
    I totally disapprove use the anonymity to shouting insults, irony or to force a person look ridiculous or something, which seems to have been the case.
    I want to convey to you that many of us simple users who have tried to upgrade our linux systems by trial/error, either by curiosity or need, using your experience, your knowledge and even your mistakes, because only do not make mistakes those who do not try.
    So I would like to take this opportunity to also thanking you for all you has shared with us, and inviting you, if you so wish, to participate collaboratively with your knowledge in this topic (http://ubuntuforum-pt.org/index. php? topic = 29799.msg421708 # msg421708) whose only difficulty is to be writen in Portuguese.
    The topic was created by this user (http://ubuntuforum-pt.org/index.php?action=profile u = 18 026) which since 2007 or before has sought to share their knowledge with anyone interested to compile a linux kernel (specifically in ubuntu/debian systems) that would providing a balance between performance and stability.
    Thanking you for your generous attention and I apologize for the long post!
    Reggards.

    ReplyDelete
  24. I would like to say that dismissing this patch as "only helping developers" is short-sighted. Yes, I know there are many Linux users that never open a terminal, every application is clicked off a menu within X. At the other extreme are the guys like Linus that mostly use processes in the terminal, and X processes are rare afterthoughts.

    What I submit, though, is that there are a lot of people in the middle that would benefit from this patch (or equivalent userspace, systemd-style solution). Many linux users use their computers for work and for recreation, and there are activities in both those use cases that are terminal based. Examples of terminal activities would include file-management activities (backups, source-control activities) media-encoding (ripping, transcoding, moving large files from one system to another), database activities, encryption tasks, torrenting (or other large-file copying), etc, etc.

    My point is this -- during the day, as a developer, this patch would help, no question. But in my free time, just doing random stuff that has nothing to do with development, I also hit this interactivity problem ALL THE TIME. I don't think I'm alone in this.

    I think big-time kernel hackers have an incorrect perception of the linux user base. They see two groups: kernel hackers and 'everyone else' -- helpless newbs that can barely get their fingers out of their butts long enough to click a link on a web page. In between those extremes, however, are a class of users that routinely mix terminal and X tasks and have corresponding desktop interaction issues, and the scheduling patch would be a huge improvement.

    For the record, I think the userspace approach is the more sensible approach, but what do I know ... I keep getting my fingers stuck in my butt.

    ReplyDelete
  25. I take what you're saying, but my point is not that this patch is useless. My point is that this is an heuristic tweak, and like all heuristic tweaks, it is prone to failure and corner cases. Not only that, but they're getting so excited by this that they're trying to add yet more heuristics which lead to more possible failures and corner cases. That's exactly what I was trying to avoid in creating a fair scheduler - avoid heuristics and have deterministic reliable behaviour all the time instead of excellent behaviour most of the time and sometimes abysmal failure. I've already read reports of people having regressions with Amarok playback skipping and this TTY patch. Even if you don't use Amarok itself, it's a marker that other things will be affected too.

    ReplyDelete
  26. So with the current batch of distro's that use heavily efficient repositories and repository manager applications handling when I used to compile. If I still use my server fairly heavily as a general server (SSH, FTP, Apache/PHP, Tomcat, MySQL, Mongo, transcoding) and a desktop system (Gnome, KDE and enlightenment), will I notice this while my background applications are eating away at my time or will it still appear sluggish during load as usual?

    ReplyDelete
  27. Anonymous: "Funny that whenever Con comes up with some scheduler idea (whether good or bad), the kernel guys will come up with a copy cat implementation a week after."

    SO true...

    CK: BFS. They: CFS!

    CK: Task groups. They: "miracle-200-line-patch" for task groups!

    ReplyDelete
  28. "Funny that whenever Con comes up with some scheduler idea (whether good or bad), the kernel guys will come up with a copy cat implementation a week after."

    Agreed. Linus can be such a j-erk sometimes. It can not be easy to argue with Linus. I understand Con's trouble he had with Linus.

    ReplyDelete
  29. I can't use BFS for one damned reason: after second suspend of my T500 + catalyst, xorg becomes soo slow, even mouse lags...
    I tried to write to CK, but he thought that the reason for this behavior is TuxOnIce, which is just a hibernate thing...
    I have tried BFS for at least 5 times and get xorg going slow, I'll try it again when I'll have free time to see whether it's improved in this regard...

    ReplyDelete
  30. Hi. I am the CK you spoke of.

    The mechanism that involves hibernating and resuming is absolutely entirely dependent on how the scheduler is coded for the ability to offline CPUs, break affinity and putting tasks to sleep, and then wake them up again. It is NOT a trivial thing to "just hibernate", as the hibernation mechanism and the scheduler are intimately linked. I spend a lot of time each kernel release to ensure it still hibernates with the in-kernel mechanism, but I also do have limited time at my disposal, and am only one person maintaining an entire out-of-kernel CPU scheduler. I do not have the time to test if it works with tuxonice as that is not part of the kernel normally. So I cannot ensure that tuxonice will work with BFS after it's patched in as well.

    ReplyDelete
  31. Thanks for reply, I answered to previous comments, that's why I said "write to CK" ;)

    The thing with hibernate (suspend to DISK) is NOT an issue for me, thing with suspend (suspend to RAM) IS... I'm not an expert but TOI is not connected to suspend to RAM and that is exactly what is failing for me...
    As I said, I'm trying BFS from time to time and will do it when I upgrade kernel or new BFS version will come out...
    I really appreciate Your work, but I can't use it at the moment for my config, when I'll be able, I definitely will send You a mail or post here...

    Cheers :)

    ReplyDelete
  32. You're such a drama queen, Con.

    ReplyDelete
  33. If somebody is still reading here:
    Could this kernel option
    /proc/sys/kernel/sched_autogroup_enabled
    result in a performance decrease of CPU intensive jobs and benchmarks like super-pi by about 30%?
    I observe this on a Core i7-980 with a 2.6.37 kernel, when comparing with a Core i7 960 with older kernel.

    ReplyDelete
  34. Yes it could, provided you make sure to confirm it occurs with it on and doesn't occur with it off.

    ReplyDelete
  35. Thanks for this informative post! I really appreciated it :)

    ReplyDelete
  36. I want the kind of responsiveness that your patch enables, more than I want gnome or mplayer. ;)

    ReplyDelete
  37. I want the kind of responsiveness that your patch enables, more than I want gnome or mplayer. ;)

    ReplyDelete