Dual core madness; Make -j never works

[toadlife]

I've had my dual core system for a couple of weeks now, and one thing I have noticed is that with most stuff, you can't fully utilize your processor when compiling. In FreeBSD you can specifify multiple jobs in makle when doing make buildworld/buildkernel and both cores are used. The results in uber fast compilation. Unfortunately, with almost all ports, specifying multiple jobs with make (e.g. `make -j3 install`) breaks compilation.

Anyone else here have this problem with other package management systems - like portage in gentoo? It sucks to have only 50% of your processor used when compiling large applications.

[Jack2000]

can not you fool the system
it has only one cpu but cpu = cpu0+ cpu1
??

[toadlife]

can not you fool the system
it has only one cpu but cpu = cpu0+ cpu1
??

No. I could install the non-SMP kernel, but then only one core in my processor would be used. That would handicap my machine when running programs that support SMP.

[Orethrius]

No. I could install the non-SMP kernel, but then only one core in my processor would be used. That would handicap my machine when running programs that support SMP.

 What about installing and configuring distcc?  Seems to me that an internal distribution of compiles across the cores would come in handy.

[toadlife]

What about installing and configuring distcc?  Seems to me that an internal distribution of compiles across the cores would come in handy.

INteresting idea. I'll look into that.

[toadlife]

distcc wouldn't work because it uses the same -j switch that regular make does.

After going over what I know about the FreebSD ports systems, I know why it doesn't work now. FreeBSD uses 'make' to do every step of port installation.

When you are in a ports directory there are four main steps in installing a port. I might be leaving steps out here, but this is the jist of it...

1) make fetch
2) make configure
3) make build
4) make install

If you type 'make install', it actually implies, `make fetch configure build install'

If you say 'make install -j2', then make will try to do the first two jobs at the same time - fetch and configure. Ooops! That can't work because you have to fetch what you are configuring first.

Hopefully port makers can override the -j setting in their ports or the FreeBSD dev team can fix the ports system to override the -j switch until the compilation actually starts.

[Orethrius]

So it sounds like the BSD installer is setup to symlink to a bunch of other functions - why not configure it to pass the -j2 switch to 'make build' only; or, failing that, make your own installer script?  It doesn't sound like it'd be that difficult.

[toadlife]

No everything is done with 'make', thus the makefile cotrols the entire process. I've talked to some other BSD users and as far as they know the only way to fix it is to fix the ports induvidually.

Oh well.

[KernelPanic]

Moving to Tech Support.

[toadlife]

Moving to Tech Support.

Hmmm, I wasn't asking for help with this but...ok. I just wanted to know if other package management systems that compiled programs had this problem.



Anyway, I think I figured it out. You can set `MAKE_ARGS=-j3` in your /etc/Make.conf and the -j switch only activates on build part of installing a port. So far it seems to work.