Transporting MJ over a long distance (Mindcrack)

[Abdiel]

The whole "loading chunks" concept is frankly utterly ridiculous, and I only hope that one day a competent programmer joins Mojang and we can get rid of it once and for all.

 

[Hydra]

But Abdiel: how do you propose the engine would handle infinite terrain then?

 

[hotblack desiato]

the chunk-concept is okay. but why not 3d-chunks? this way, the world anchor would just load what's needed (it's completely irrelevant what happens 100 blocks above or 60 blocks below the track.

 

[Poppycocks]

the chunk-concept is okay. but why not 3d-chunks? this way, the world anchor would just load what's needed (it's completely irrelevant what happens 100 blocks above or 60 blocks below the track.

I think that FlowerChild explained this best:

Giving Steve an artificial block to bypass a technical game limitation is even more jarring. It's providing an in-game object that makes no sense whatsoever except within the context of bypassing a technical limitation, and puts the player in the weird position of playing some kind of meta-game within the game that involves balancing chunk load vs performance, and whose proper usage requires the player become somewhat of an expert in the way the chunk-loading system works behind the scenes.

To me, I'd much rather just leave it at "when you're far away, stuff stops working", than make something like that part of "proper" game-play. To me, it's no real solution, just a band-aid that places the onus on the player to manage chunk-loading themselves.

I'd challenge you to find one commercial game that does something like that man, whereas I can think of many where stuff stops working at a distance. Generally, as a professional game-designer, I aspire to always provide commercial-quality features that feel like a natural part of the game. This isn't such a feature.

As far as 3d loaders go, technical limitations won't allow for this. As a chunk is an area of 16x16x256 blocks. You either load the whole chunk, or nothing. It'd be nice if minecraft had cubic chunks and only loaded visible chunks, but that probably won't happen ever. A man can dream though, right?

 

[Abdiel]

But Abdiel: how do you propose the engine would handle infinite terrain then?

We actually had a quite in-depth discussion about this with a few friends (also programmers like me) about half a year ago. I don't remember all the details here on the spot, but here are the most significant bits:

- Decouple static data ("world") from dynamic "entities". Assume that World almost never changes, if it does, only as a result of some Entity and only on a miniature scale (few blocks at a time).
- World resides in a database. Entities are loaded or unloaded into memory. The only time pieces of World are loaded into the server's memory at all is when it is being changed (+- some collision detection). There is absolutely no reason to keep hundreds of chunks of static solid rock in memory, ever.
- When client explores an area, send them updated World data once. From there on only the client takes care of rendering World, the server doesn't need to care about it.

This alone would reduce the amount of data that the server needs to store in memory at one time by a few orders of magnitude. Let's look at Entities next:

- Every Entity (in vanilla) has a very simple and predictable behavior. A furnace smelts one block every 10 seconds, a minecart moves 10 blocks a second, a repeater sends a redstone signal four ticks after it receives one. Forgetting redstone for a while (I'll get to that in a bit), we can predict the state of any one entity in any moment, if we know its state now.
- Simulate running entities only when a client is around to observe them. What exactly this means depends on what's computationally feasible, but also on the type of the entity. Worst case, simulate an entity whenever a client is in a given range.
- Example: A minecart should be moving whenever a player looks at it. But a furnace doesn't need to be smelting while a player is not looking at the GUI. It is enough to remember when the smelting started, and how many items and fuel are there in the furnace. Whenever a player opens the GUI, it is trivial (as in, a few instructions) to calculate what state the smelting is in.
- Entities sometimes cause various actions to happen, we'll call that Triggers. A furnace that finishes smelting changes its texture. A plant grows. Etc.
- We may need to update the game state (and World in some cases) whenever a Trigger happens regardless of whether an Entity is being actively simulated. This can be easily done by a technique called Discrete Simulation (look it up, or just trust me that it's simple).
- Whenever a client comes within range or starts observing an Entity, the Entity may need to "catch up" with its internal state, if it was not being simulated. E.g. as I mentioned before, the furnace needs to update the number of smelted items. For most vanilla entities, if we know when was the entity last simulated, and what state we left it off, we can calculate the state it is in easily. (Again for a furnace, take the time since the last simulation, divide by 10 seconds, subtract that number of inputs and add that number of outputs +- some math with fuel.)
- The simplest, but computationally most expensive form of catching up, is to simply simulate the entity for the number of ticks it had been unobserved. Again, a discrete simulation ensures that we don't actually have to perform as many actual computations even in this case.

The last thing that's being a problem is redstone, as it is easily capable of emitting loads of updates if left running. Our collective opinion was that we should optimize for small systems, with up to few hundred redstone blocks, and not for people building fully functional computers and stuff (sorry.) Unfortunately, there is not much that can be optimized here. You can treat an uninterrupted line of redstone as a single entity. There are a few known algorithms for identifying patterns, such as "understanding" what a redstone clock is, but those are for the most part theoretical and not easily applicable to such a complex situation. However, we could modify game mechanics a little: add simple circuits as individual blocks, like Redpower does. These circuits are much more predictable than a random mess of wires and torches, and we could perform the "catch up" on them without having to simulate every tick.

So, to sum up, the server would keep in memory and work with:

- Small pieces of World, mostly for collision detection and performing updates. Think, uh, a 8^3 or 16^3 area around every client, less than that for mobs. (In practice this would actually probably be more to speed up hard drive access, but it would be 3D areas, not stupidly tall chunks.)
- New pieces of World that are being freshly generated or sent to clients.
- Entities that are currently being observed by clients, which are performing some action (minecarts) and are not just waiting for a Trigger (growing reeds).
- Some specific complex behaviors for which we can't perform the "catch up" in reasonable time - i.e. complex redstone circuitry, flowing water, etc.
- (The above can be sped up by heuristically grouping entities by location, and performing many ticks for each location at once. It is much better, performance-wise, to run one area for 100 ticks, then another, independent area for 100 ticks, than trying to simulate 100 ticks in both areas once. As long as all the required simulation is done by the time the entities are observed, when exactly we calculate the changes doesn't matter.)

For a vanilla game, this reduces the load on a server by a factor of thousands. We could easily run an entire vanilla world like this, regardless of whether people are online or around any machinery.

Now, mods throw a huge wrench in all of this by being able to run practically any code with the machines. When we were talking about this, we didn't consider mods at all. With arbitrary code running in the machines, certain parts of the game would have to be constantly loaded, as there is no way to do the catch-up without simulating every tick. A (partial) solution is to provide an API to give modders a chance to add this functionality. We know that a macerator processes an item in X ticks. We could analyze simple and certain complex pipe networks and machine automation in terms of graph flows (again, look it up). But integrating all the current mods' functionality into this framework would certainly require a lot of effort and probably also re-thinking of basic concepts. The big question is (and I do not have an answer to it), whether the computation resources saved by not keeping World in memory, and running incremental updates and discrete simulations of machines whenever possible would be enough to allow us to simply run all the remaining stuff continuously.


Obviously this whole post is just a bunch of random ideas I remember from a conversation long ago. It is not intended to be anything close to a complete concept, much less any practical suggestion for an architectural change. There are huge holes in it, there are questions for which I do not have answers. But the point remains: Minecraft started as Notch's personal little project to demonstrate his coding skills. The core mechanics were not designed with best programming practices in mind, nor thinking about all the crazy stuff modders do with it these days. Looking back at it now, there are many places where huge improvements could be made. But with the heaps of code and basic designs that would have to be changed, and more code in mods that would have to adapt to this, making such a huge change is almost unthinkable. There is some low-hanging fruit to be picked up (more granular "chunks", not loading static World when not needed, discrete simulation of simple machinery), and that's mostly what I had in mind in my first post.




Please forgive the little off-topic rant, and the wall of text. I really do not mind if you stopped reading somewhere in the middle.

 

[Hydra]

I actually wrote a minecraft viewer a couple of years ago (just to fiddle with OpenGL a bit in Java) and I think you're underestimating the problem here. The game loads X chunks around a player to simulate that 'world'. Stuff like plants growing, tree leaves decaying, etc., all happens only in the chunks that are being simulated. And a lot of the optimizations that you mention are already happening. For example: teh game already has a task system (for stuff like smelting) where the game simply knows that a smelding action is going to be completed in X milliseconds. It's not checking very tick or something.

The main reason minecraft takes so much memory is simply that every chunk you can see has to be in memory and has to be translated from 'normal' memory to GPU memory so it can be drawn. Any chunk that is visible HAS to be in memory, there is no way around that. And every chunk where something happens needs to be translated from it's memory representation to the vertex representation in GPU memory, and that's simply a lot of work for both the CPU and the memory bus.

I'm not saying Notch is the best programmer ever but I think you're underestimating how different a game like MC is from something like a 3D shooter.

 

[Plscks]

I tried to come up with a solution for you, and this is my current setup:

Spoiler

the gate at the charging station cell is set to "is full", the filter at the charging station has an empty cell in it. The gate above the quarry cell is set to "is empty" and the filter has a full cell.

The turtle is an engineering turtle (crescent hammer + turtle)
The code is:

Spoiler

while true do
  while true do
  os.pullEvent("redstone")
  if redstone.getInput("left") then break end
  sleep(1)
end
turtle.attack()
turtle.suck()
turtle.turnLeft()
turtle.turnLeft()
turtle.drop(1)
turtle.turnLeft()
turtle.turnLeft()
end

Make sure to either name it "startup" or have it run from a different startup script. This way, you won't have to restart the turtle after a server restart / turning ssp off+on

I haven't tested this thoroughly, but it seems to work just fine atm .

By the way, it's perfectly possible to have another turtle move the quarry and everything after the quarry's done.

I love this and this turtle program and setup has worked for me and my remote quarry, thank you!

 

[Poppycocks]

I love this and this turtle program and setup has worked for me and my remote quarry, thank you!

No problem! I'm just glad it works for you.

 

[Someone Else 37]

<Wall of Text>

One thing you didn't address is when/how worlds are generated serverside. I got the impression that you assumed that the entire world had already been generated by the time the client gets to see any of it. This prevents the server needing to generate terrain during runtime, but leads to problems of its own. First, an infinite world would take infinite disk space, nearly all of which nobody cares about. Second, it would require infinite time or processing power to generate at all. Third, infinite terrain would include infinite opportunities for, say, water springs to flow. My impression, however, could well be completely wrong,

From what I gather, you're trying to split the 16*16*256 chunks that we have now into a gazillion of 1 meter cubed chunks, and optimizing the heck out of as much vanilla stuff as possible. I see two problems here: Although I'm no programmer, having the server send a few tens of chunks seems a lot more efficient than sending hundreds of thousands of individual blocks (although there is certainly room for optimization). Second, mods wanting to do things with bits you had thought could be optimizes away. For example: You describe a furnace that only updates itself when it runs out of fuel or smeltables, or when someone opens its GUI. Now, what if I want to make a mod that can pull products out of a furnace after it smelts each one? I'd have to either go in and remove at least some of your optimization, or try to get you to do the same thing.

Also note that if Mojang did happen to pick up a programmer that wanted to implement all this, people would not be happy while development ceased for a few years as the entire code was re-written.

 

[billbertking1]

Could use Tesseracts... That's the easiest and fastest, and don't have to worry about laying down anything. Just place one at the power station, another at the quarry, and set them to the same frequency that's not taken. Then you're done!

 

[hotblack desiato]

the thread was started, when tesseracts weren't available, and since they have a 25% loss, they are still not the best way to do it...

 

[billbertking1]

the thread was started, when tesseracts weren't available, and since they have a 25% loss, they are still not the best way to do it...

Oh ok, I didn't know they had a 25% loss too, and that they weren't around then...

 

[Mash]

This is why i usually refrain from posting on old threads like this. Issues that were issues back then usually aren't issues anymore.

 

[Poppycocks]

That's why it's usually frowned upon to post on old threads like this. Issues what were issues back then are rarely issues anymore and all new communication is completely off.

 

[Fuzzlewhumper]

Did anyone mention using re-batteries or other EU storage medium and ender chests?
Also Bluelectricity batteries can be moved about that way as well.
Also liquid transposers for using cans/cells for the liquid fuels via ender chest.
All of that is mostly loss less.

In your case, it's biomass right? If you don't want to use tesseracts for that, the cells and cans method works too. Need chunk loaders though. I like spot loaders personally. But that's just me.

Oh and necro-posts for the win.

 

[ryter78]

Since tesseracts are already in the game, be creative, use tankcarts and the nether.

 

[Poppycocks]

Oh Chesus H tapdancing Christ.

 

[twisto51]

Well, since the thread has been revived, and we're talking about tesseracts, don't forget that the loss is only on energy. You can still send liquids and items, like for instance, steam, batteries/recs without loss. I've been having fun making tesseracts work in many of the places I used to use enderchests.

 

[whizzball1]

Why not use Tesseracts?

 

[Zivel]

I think that convoluted and crazy contraption you guys have made is great and all but Tesseracts are so much easier. Hell if you dont want to suffer the power loss do the magma crucible and magmatic engines and Tess across the lava.